[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:16:07 dist-test-slave-4cx3 krb5kdc[28396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:16:07 dist-test-slave-4cx3 krb5kdc[28396](info): set up 2 sockets
Sep 29 20:16:07 dist-test-slave-4cx3 krb5kdc[28396](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 20:16:08.063124 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.008s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:16:08 dist-test-slave-4cx3 krb5kdc[28396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:16:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:16:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
I20260929 20:16:08.235982 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:35841
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:32999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:35841
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:08.557735 28409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:08.558035 28409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:08.558153 28409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:08.565506 28409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:16:08.565625 28409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:08.565737 28409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:08.565838 28409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:16:08.565928 28409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:16:08.576105 28409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:32999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:35841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:35841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:08.577489 28409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:08.578815 28409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:08.590942 28414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:08.591008 28409 server_base.cc:1034] running on GCE node
W20260929 20:16:08.590894 28415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:08.590921 28417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:08.592177 28409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:08.593572 28409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:08.594789 28409 hybrid_clock.cc:648] HybridClock initialized: now 1790712968594715 us; error 52 us; skew 500 ppm
I20260929 20:16:08.595221 28409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:08 dist-test-slave-4cx3 krb5kdc[28396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712968, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:08.598359 28409 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:16:08.599936 28409 webserver.cc:460] Webserver started at http://127.27.185.126:41947/ using document root <none> and password file <none>
I20260929 20:16:08.600627 28409 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:08.600771 28409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:08.601081 28409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:08.604168 28409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "095f73bd6c654d21ae6e31fbe5b4a540"
format_stamp: "Formatted at 2026-09-29 20:16:08 on dist-test-slave-4cx3"
server_key: "3f23ed1d6303db5f01daf3b4f63b066c"
server_key_iv: "e1689fe47eec649f9cfd2b8c04a69e1f"
server_key_version: "encryptionkey@0"
I20260929 20:16:08.604960 28409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "095f73bd6c654d21ae6e31fbe5b4a540"
format_stamp: "Formatted at 2026-09-29 20:16:08 on dist-test-slave-4cx3"
server_key: "3f23ed1d6303db5f01daf3b4f63b066c"
server_key_iv: "e1689fe47eec649f9cfd2b8c04a69e1f"
server_key_version: "encryptionkey@0"
I20260929 20:16:08.610225 28409 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:16:08.613726 28424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:08.615029 28409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 20:16:08.615216 28409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "095f73bd6c654d21ae6e31fbe5b4a540"
format_stamp: "Formatted at 2026-09-29 20:16:08 on dist-test-slave-4cx3"
server_key: "3f23ed1d6303db5f01daf3b4f63b066c"
server_key_iv: "e1689fe47eec649f9cfd2b8c04a69e1f"
server_key_version: "encryptionkey@0"
I20260929 20:16:08.615463 28409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:08.653506 28409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:08.658217 28409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:08.658540 28409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:08.677459 28409 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:35841
I20260929 20:16:08.677484 28476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:35841 every 8 connection(s)
I20260929 20:16:08.678846 28409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:16:08.682544 28477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:08.682821 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 28409
I20260929 20:16:08.683143 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:16:08.684988 28389 external_mini_cluster.cc:1428] Setting key 1509c7374929f1752bf0d99edc112c46
I20260929 20:16:08.693698 28477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 095f73bd6c654d21ae6e31fbe5b4a540: Bootstrap starting.
I20260929 20:16:08.696668 28477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 095f73bd6c654d21ae6e31fbe5b4a540: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:08.697670 28477 log.cc:824] T 00000000000000000000000000000000 P 095f73bd6c654d21ae6e31fbe5b4a540: Log is configured to *not* fsync() on all Append() calls
I20260929 20:16:08.700726 28477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 095f73bd6c654d21ae6e31fbe5b4a540: No bootstrap required, opened a new log
I20260929 20:16:08.707957 28477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 095f73bd6c654d21ae6e31fbe5b4a540 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095f73bd6c654d21ae6e31fbe5b4a540" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 35841 } }
I20260929 20:16:08.708379 28477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 095f73bd6c654d21ae6e31fbe5b4a540 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:08.708491 28477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 095f73bd6c654d21ae6e31fbe5b4a540 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 095f73bd6c654d21ae6e31fbe5b4a540, State: Initialized, Role: FOLLOWER
I20260929 20:16:08.709148 28477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 095f73bd6c654d21ae6e31fbe5b4a540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095f73bd6c654d21ae6e31fbe5b4a540" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 35841 } }
I20260929 20:16:08.709406 28477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 095f73bd6c654d21ae6e31fbe5b4a540 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:16:08.709573 28477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 095f73bd6c654d21ae6e31fbe5b4a540 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:16:08.709757 28477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 095f73bd6c654d21ae6e31fbe5b4a540 [term 0 FOLLOWER]: Advancing to term 1
Sep 29 20:16:08 dist-test-slave-4cx3 krb5kdc[28396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:08.712253 28477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 095f73bd6c654d21ae6e31fbe5b4a540 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095f73bd6c654d21ae6e31fbe5b4a540" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 35841 } }
I20260929 20:16:08.712877 28477 leader_election.cc:302] T 00000000000000000000000000000000 P 095f73bd6c654d21ae6e31fbe5b4a540 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 095f73bd6c654d21ae6e31fbe5b4a540; no voters:
I20260929 20:16:08.713599 28477 leader_election.cc:288] T 00000000000000000000000000000000 P 095f73bd6c654d21ae6e31fbe5b4a540 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:16:08.713719 28482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 095f73bd6c654d21ae6e31fbe5b4a540 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:16:08.715595 28482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 095f73bd6c654d21ae6e31fbe5b4a540 [term 1 LEADER]: Becoming Leader. State: Replica: 095f73bd6c654d21ae6e31fbe5b4a540, State: Running, Role: LEADER
I20260929 20:16:08.716156 28482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 095f73bd6c654d21ae6e31fbe5b4a540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095f73bd6c654d21ae6e31fbe5b4a540" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 35841 } }
I20260929 20:16:08.716351 28477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 095f73bd6c654d21ae6e31fbe5b4a540 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:16:08.718784 28484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 095f73bd6c654d21ae6e31fbe5b4a540 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 095f73bd6c654d21ae6e31fbe5b4a540. Latest consensus state: current_term: 1 leader_uuid: "095f73bd6c654d21ae6e31fbe5b4a540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095f73bd6c654d21ae6e31fbe5b4a540" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 35841 } } }
I20260929 20:16:08.719209 28484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 095f73bd6c654d21ae6e31fbe5b4a540 [sys.catalog]: This master's current role is: LEADER
I20260929 20:16:08.727974 28487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:16:08.727840 28483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 095f73bd6c654d21ae6e31fbe5b4a540 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "095f73bd6c654d21ae6e31fbe5b4a540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095f73bd6c654d21ae6e31fbe5b4a540" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 35841 } } }
I20260929 20:16:08.728173 28483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 095f73bd6c654d21ae6e31fbe5b4a540 [sys.catalog]: This master's current role is: LEADER
I20260929 20:16:08.730408 28479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:08.688775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.27.185.126:35841)
0929 20:16:08.689575 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:08.689586 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:08.690743 (+ 1157us) server_negotiation.cc:407] Connection header received
0929 20:16:08.699722 (+ 8979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:08.699745 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:08.700182 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:08.700555 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:08.703447 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:08.704753 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:08.706410 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:08.707130 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:08.707334 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:08.713780 (+ 6446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:08.713795 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:08.713818 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:08.713879 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:08.719599 (+ 5720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:08.721217 (+ 1618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:08.721229 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:08.721240 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:08.721517 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:08.722274 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:08.722281 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:08.722287 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:08.723199 (+ 912us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:08.727298 (+ 4099us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:08.727700 (+ 402us) server_negotiation.cc:299] Negotiation successful
0929 20:16:08.728521 (+ 821us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":389,"threads_started":1}
I20260929 20:16:08.733740 28487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:16:08.744050 28487 catalog_manager.cc:1347] Generated new cluster ID: 45999c6b24eb4174bbd846c8e8d94064
I20260929 20:16:08.744174 28487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:16:08.776456 28487 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:16:08.777829 28487 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:16:08.793201 28487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 095f73bd6c654d21ae6e31fbe5b4a540: Generated new TSK 0
I20260929 20:16:08.794432 28487 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
I20260929 20:16:08.855664 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:0
--local_ip_for_outbound_sockets=127.27.185.65
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:35841
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:32999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:09.178879 28504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:09.179165 28504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:09.179260 28504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:09.186278 28504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:09.186453 28504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:09.186647 28504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:16:09.196372 28504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:32999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:35841
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:09.197705 28504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:09.199077 28504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:09.212512 28510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:09.212543 28512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:09.212422 28509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:09.213534 28504 server_base.cc:1034] running on GCE node
I20260929 20:16:09.214154 28504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:09.215085 28504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:09.216250 28504 hybrid_clock.cc:648] HybridClock initialized: now 1790712969216191 us; error 59 us; skew 500 ppm
I20260929 20:16:09.216696 28504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:09 dist-test-slave-4cx3 krb5kdc[28396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712969, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:09.220199 28504 init.cc:377] Logged in from keytab as kudu/127.27.185.65@KRBTEST.COM (short username kudu)
I20260929 20:16:09.221707 28504 webserver.cc:460] Webserver started at http://127.27.185.65:33999/ using document root <none> and password file <none>
I20260929 20:16:09.222376 28504 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:09.222530 28504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:09.222851 28504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:09.225767 28504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-0/data/instance:
uuid: "1504578c6ea24a6aba79baed4d386b5f"
format_stamp: "Formatted at 2026-09-29 20:16:09 on dist-test-slave-4cx3"
server_key: "b03ed4100b840ce76ae301ca44c74e24"
server_key_iv: "61be8611ad5026eae27b202562e625c8"
server_key_version: "encryptionkey@0"
I20260929 20:16:09.226564 28504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance:
uuid: "1504578c6ea24a6aba79baed4d386b5f"
format_stamp: "Formatted at 2026-09-29 20:16:09 on dist-test-slave-4cx3"
server_key: "b03ed4100b840ce76ae301ca44c74e24"
server_key_iv: "61be8611ad5026eae27b202562e625c8"
server_key_version: "encryptionkey@0"
I20260929 20:16:09.231813 28504 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260929 20:16:09.235114 28519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:09.236522 28504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:16:09.236701 28504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "1504578c6ea24a6aba79baed4d386b5f"
format_stamp: "Formatted at 2026-09-29 20:16:09 on dist-test-slave-4cx3"
server_key: "b03ed4100b840ce76ae301ca44c74e24"
server_key_iv: "61be8611ad5026eae27b202562e625c8"
server_key_version: "encryptionkey@0"
I20260929 20:16:09.236900 28504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:09.268874 28504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:09.273519 28504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:09.273834 28504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:09.274854 28504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:16:09.276283 28504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:16:09.276383 28504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:09.276546 28504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:16:09.276618 28504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:09.310467 28504 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:41125
I20260929 20:16:09.310550 28632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:41125 every 8 connection(s)
I20260929 20:16:09.311841 28504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:16:09.312647 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 28504
I20260929 20:16:09.312906 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance
I20260929 20:16:09.313650 28389 external_mini_cluster.cc:1428] Setting key 9a14fe3a21ae26cd40c92be06eed640e
Sep 29 20:16:09 dist-test-slave-4cx3 krb5kdc[28396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712969, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.185.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.66@KRBTEST.COM" created.
I20260929 20:16:09.335656 28635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:09.314777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:37521 (local address 127.27.185.126:35841)
0929 20:16:09.315233 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:09.315238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:09.316601 (+ 1363us) server_negotiation.cc:407] Connection header received
0929 20:16:09.319648 (+ 3047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:09.319652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:09.319718 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:09.319824 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:09.321746 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:09.323043 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:09.324638 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:09.325397 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:09.325538 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:09.329310 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:09.329318 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:09.329324 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:09.329370 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:09.332662 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:09.333391 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:09.333395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:09.333398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:09.333520 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:09.333874 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:09.333878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:09.333880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:09.334201 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:09.334345 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:09.335147 (+ 802us) server_negotiation.cc:299] Negotiation successful
0929 20:16:09.335328 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":261,"threads_started":1}
I20260929 20:16:09.337137 28636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:09.315276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:35841 (local address 127.27.185.65:37521)
0929 20:16:09.316384 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:09.316492 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:09.319289 (+ 2797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:09.320097 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:09.320111 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:09.320910 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:09.321556 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:09.321574 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:09.323193 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:09.323197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:09.324496 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:09.324508 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:09.325598 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:09.325603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:09.325790 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:09.326466 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:09.326493 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:09.329059 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:09.332790 (+ 3731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:09.332796 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:09.332808 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:09.333216 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:09.333615 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:09.333618 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:09.333620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:09.333779 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:09.334313 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:09.334319 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:09.334688 (+ 369us) client_negotiation.cc:769] Sending connection context
0929 20:16:09.334972 (+ 284us) client_negotiation.cc:240] Negotiation successful
0929 20:16:09.335569 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"thread_start_us":482,"threads_started":1}
I20260929 20:16:09.338740 28633 heartbeater.cc:347] Connected to a master server at 127.27.185.126:35841
I20260929 20:16:09.339066 28633 heartbeater.cc:464] Registering TS with master...
I20260929 20:16:09.339890 28633 heartbeater.cc:511] Master 127.27.185.126:35841 requested a full tablet report, sending...
I20260929 20:16:09.342353 28442 ts_manager.cc:194] Registered new tserver with Master: 1504578c6ea24a6aba79baed4d386b5f (127.27.185.65:41125)
I20260929 20:16:09.344599 28442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.65@KRBTEST.COM'} at 127.27.185.65:37521
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
I20260929 20:16:09.359472 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:0
--local_ip_for_outbound_sockets=127.27.185.66
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:35841
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:32999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:09.675475 28639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:09.675763 28639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:09.675952 28639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:09.683174 28639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:09.683342 28639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:09.683558 28639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:16:09.693055 28639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:32999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:35841
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:09.694325 28639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:09.695667 28639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:09.708784 28644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:09.708745 28645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:09.708853 28648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:09.709631 28639 server_base.cc:1034] running on GCE node
I20260929 20:16:09.710194 28639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:09.711107 28639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:09.712338 28639 hybrid_clock.cc:648] HybridClock initialized: now 1790712969712239 us; error 61 us; skew 500 ppm
I20260929 20:16:09.712663 28639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:09 dist-test-slave-4cx3 krb5kdc[28396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712969, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:09.716223 28639 init.cc:377] Logged in from keytab as kudu/127.27.185.66@KRBTEST.COM (short username kudu)
I20260929 20:16:09.717941 28639 webserver.cc:460] Webserver started at http://127.27.185.66:43825/ using document root <none> and password file <none>
I20260929 20:16:09.718681 28639 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:09.718793 28639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:09.719126 28639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:09.722389 28639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-1/data/instance:
uuid: "2d8510de3e034072b9bfb07234322609"
format_stamp: "Formatted at 2026-09-29 20:16:09 on dist-test-slave-4cx3"
server_key: "99d70cd3c7604842078f27d27e0f82d2"
server_key_iv: "2442292f5ada0ccd538280664f21d64b"
server_key_version: "encryptionkey@0"
I20260929 20:16:09.723235 28639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance:
uuid: "2d8510de3e034072b9bfb07234322609"
format_stamp: "Formatted at 2026-09-29 20:16:09 on dist-test-slave-4cx3"
server_key: "99d70cd3c7604842078f27d27e0f82d2"
server_key_iv: "2442292f5ada0ccd538280664f21d64b"
server_key_version: "encryptionkey@0"
I20260929 20:16:09.728802 28639 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 20:16:09.732844 28655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:09.734246 28639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 20:16:09.734436 28639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "2d8510de3e034072b9bfb07234322609"
format_stamp: "Formatted at 2026-09-29 20:16:09 on dist-test-slave-4cx3"
server_key: "99d70cd3c7604842078f27d27e0f82d2"
server_key_iv: "2442292f5ada0ccd538280664f21d64b"
server_key_version: "encryptionkey@0"
I20260929 20:16:09.734683 28639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:09.758494 28639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:09.762813 28639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:09.763072 28639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:09.764074 28639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:16:09.765442 28639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:16:09.765522 28639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:09.765610 28639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:16:09.765658 28639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:09.801169 28639 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:44091
I20260929 20:16:09.801246 28768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:44091 every 8 connection(s)
I20260929 20:16:09.802538 28639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:16:09.805603 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 28639
I20260929 20:16:09.805796 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance
I20260929 20:16:09.806231 28389 external_mini_cluster.cc:1428] Setting key b3fd26f9ed4a62682da50df85425a8f8
Sep 29 20:16:09 dist-test-slave-4cx3 krb5kdc[28396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712969, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.185.67@KRBTEST.COM; defaulting to no policy
I20260929 20:16:09.825278 28635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:09.805230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:33217 (local address 127.27.185.126:35841)
0929 20:16:09.805514 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:09.805521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:09.806787 (+ 1266us) server_negotiation.cc:407] Connection header received
0929 20:16:09.809170 (+ 2383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:09.809178 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:09.809285 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:09.809397 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:09.810854 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:09.812009 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:09.814266 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:09.814680 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:09.814789 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:09.819135 (+ 4346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:09.819140 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:09.819143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:09.819220 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:09.821942 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:09.822771 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:09.822778 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:09.822786 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:09.822917 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:09.823421 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:09.823426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:09.823429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:09.823693 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:09.823855 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:09.824615 (+ 760us) server_negotiation.cc:299] Negotiation successful
0929 20:16:09.824844 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.67@KRBTEST.COM" created.
I20260929 20:16:09.826412 28771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:09.805764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:35841 (local address 127.27.185.66:33217)
0929 20:16:09.806606 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:09.806662 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:09.808886 (+ 2224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:09.809595 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:09.809603 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:09.810125 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:09.810686 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:09.810701 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:09.812161 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:09.812168 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:09.814123 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:09.814141 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:09.814919 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:09.814926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:09.815265 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:09.816079 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:09.816126 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:09.818880 (+ 2754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:09.822107 (+ 3227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:09.822117 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:09.822135 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:09.822643 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:09.823037 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:09.823043 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:09.823046 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:09.823288 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:09.823832 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:09.823840 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:09.824253 (+ 413us) client_negotiation.cc:769] Sending connection context
0929 20:16:09.824530 (+ 277us) client_negotiation.cc:240] Negotiation successful
0929 20:16:09.825077 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":333,"threads_started":1}
I20260929 20:16:09.828006 28769 heartbeater.cc:347] Connected to a master server at 127.27.185.126:35841
I20260929 20:16:09.828320 28769 heartbeater.cc:464] Registering TS with master...
I20260929 20:16:09.829032 28769 heartbeater.cc:511] Master 127.27.185.126:35841 requested a full tablet report, sending...
I20260929 20:16:09.830760 28442 ts_manager.cc:194] Registered new tserver with Master: 2d8510de3e034072b9bfb07234322609 (127.27.185.66:44091)
I20260929 20:16:09.831709 28442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.66@KRBTEST.COM'} at 127.27.185.66:33217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
I20260929 20:16:09.847218 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:0
--local_ip_for_outbound_sockets=127.27.185.67
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:35841
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:32999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:10.178541 28774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:10.178839 28774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:10.179015 28774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:10.186965 28774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:10.187083 28774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:10.187289 28774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:16:10.197441 28774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:32999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:35841
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:10.198761 28774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:10.200197 28774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:10.213119 28780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:10.213219 28782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:10.213356 28779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:10.214433 28774 server_base.cc:1034] running on GCE node
I20260929 20:16:10.215140 28774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:10.216220 28774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:10.217458 28774 hybrid_clock.cc:648] HybridClock initialized: now 1790712970217379 us; error 64 us; skew 500 ppm
I20260929 20:16:10.217816 28774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:10 dist-test-slave-4cx3 krb5kdc[28396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712970, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:10.221122 28774 init.cc:377] Logged in from keytab as kudu/127.27.185.67@KRBTEST.COM (short username kudu)
I20260929 20:16:10.222663 28774 webserver.cc:460] Webserver started at http://127.27.185.67:43285/ using document root <none> and password file <none>
I20260929 20:16:10.223449 28774 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:10.223610 28774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:10.223881 28774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:10.226763 28774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-2/data/instance:
uuid: "730a815523e847ec9b40decb4c0056e4"
format_stamp: "Formatted at 2026-09-29 20:16:10 on dist-test-slave-4cx3"
server_key: "8c3126ebe7bde0531574c5f14f14861d"
server_key_iv: "ca7a72a34afa8659b82752af7ffab85b"
server_key_version: "encryptionkey@0"
I20260929 20:16:10.227595 28774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance:
uuid: "730a815523e847ec9b40decb4c0056e4"
format_stamp: "Formatted at 2026-09-29 20:16:10 on dist-test-slave-4cx3"
server_key: "8c3126ebe7bde0531574c5f14f14861d"
server_key_iv: "ca7a72a34afa8659b82752af7ffab85b"
server_key_version: "encryptionkey@0"
I20260929 20:16:10.233320 28774 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:16:10.236886 28789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:10.238401 28774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:16:10.238601 28774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "730a815523e847ec9b40decb4c0056e4"
format_stamp: "Formatted at 2026-09-29 20:16:10 on dist-test-slave-4cx3"
server_key: "8c3126ebe7bde0531574c5f14f14861d"
server_key_iv: "ca7a72a34afa8659b82752af7ffab85b"
server_key_version: "encryptionkey@0"
I20260929 20:16:10.238831 28774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:10.252974 28774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:10.256835 28774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:10.257135 28774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:10.258080 28774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:16:10.259464 28774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:16:10.259555 28774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:10.259696 28774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:16:10.259760 28774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:10.293459 28774 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:46347
I20260929 20:16:10.293632 28902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:46347 every 8 connection(s)
I20260929 20:16:10.294787 28774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:16:10.303932 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 28774
I20260929 20:16:10.304149 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance
I20260929 20:16:10.304585 28389 external_mini_cluster.cc:1428] Setting key a61b0cc1cd97ca793f5eefdb653eac37
Sep 29 20:16:10 dist-test-slave-4cx3 krb5kdc[28396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712970, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:10.315264 28635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:10.297457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:56093 (local address 127.27.185.126:35841)
0929 20:16:10.297654 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:10.297659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:10.298813 (+ 1154us) server_negotiation.cc:407] Connection header received
0929 20:16:10.300635 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:10.300640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:10.300723 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:10.300810 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:10.302241 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:10.302987 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:10.304839 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:10.305523 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:10.305661 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:10.309674 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:10.309679 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:10.309682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:10.309715 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:10.312343 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:10.313045 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:10.313058 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:10.313061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:10.313182 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:10.313600 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:10.313605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:10.313609 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:10.313874 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:10.314043 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:10.314711 (+ 668us) server_negotiation.cc:299] Negotiation successful
0929 20:16:10.314929 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 20:16:10.316250 28905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:10.297919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:35841 (local address 127.27.185.67:56093)
0929 20:16:10.298652 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:10.298689 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:10.300375 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:10.301030 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:10.301038 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:10.301535 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:10.302098 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:10.302116 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:10.303124 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:10.303127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:10.304662 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:10.304697 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:10.305760 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:10.305769 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:10.306094 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:10.306779 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:10.306801 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:10.309449 (+ 2648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:10.312494 (+ 3045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:10.312503 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:10.312519 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:10.312915 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:10.313292 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:10.313297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:10.313301 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:10.313484 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:10.314036 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:10.314043 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:10.314352 (+ 309us) client_negotiation.cc:769] Sending connection context
0929 20:16:10.314606 (+ 254us) client_negotiation.cc:240] Negotiation successful
0929 20:16:10.315124 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":270,"threads_started":1}
I20260929 20:16:10.317664 28903 heartbeater.cc:347] Connected to a master server at 127.27.185.126:35841
I20260929 20:16:10.317991 28903 heartbeater.cc:464] Registering TS with master...
I20260929 20:16:10.318805 28903 heartbeater.cc:511] Master 127.27.185.126:35841 requested a full tablet report, sending...
I20260929 20:16:10.320478 28442 ts_manager.cc:194] Registered new tserver with Master: 730a815523e847ec9b40decb4c0056e4 (127.27.185.67:46347)
I20260929 20:16:10.321381 28389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:16:10.321502 28442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.67@KRBTEST.COM'} at 127.27.185.67:56093
Sep 29 20:16:10 dist-test-slave-4cx3 krb5kdc[28396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712970, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260929 20:16:10.348730 28633 heartbeater.cc:503] Master 127.27.185.126:35841 was elected leader, sending a full tablet report...
W20260929 20:16:10.432535 28389 mini_kdc.cc:317] Time spent kinit for test-user: real 0.109s user 0.001s sys 0.008s
Sep 29 20:16:10 dist-test-slave-4cx3 krb5kdc[28396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712970, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.185.65@KRBTEST.COM
I20260929 20:16:10.451710 28910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:10.435165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35222 (local address 127.27.185.65:41125)
0929 20:16:10.436219 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:10.436231 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:10.436267 (+ 36us) server_negotiation.cc:407] Connection header received
0929 20:16:10.436346 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:10.436358 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:10.436662 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:10.436841 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:10.438354 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:10.439640 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:10.441550 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:10.442189 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:10.442294 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:10.446337 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:10.446349 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:10.446363 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:10.446403 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:10.449222 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:10.449862 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:10.449869 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:10.449871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:10.449997 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:10.450336 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:10.450339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:10.450341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:10.450658 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:10.450918 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:10.451169 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 20:16:10.451358 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":856,"thread_start_us":702,"threads_started":1}
W20260929 20:16:10.452998 28547 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35222
Sep 29 20:16:10 dist-test-slave-4cx3 krb5kdc[28396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 20:16:10 dist-test-slave-4cx3 krb5kdc[28396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.65@KRBTEST.COM
I20260929 20:16:10.553046 28910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:10.537749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35224 (local address 127.27.185.65:41125)
0929 20:16:10.538018 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:10.538025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:10.538328 (+ 303us) server_negotiation.cc:407] Connection header received
0929 20:16:10.539461 (+ 1133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:10.539467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:10.539628 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:10.539749 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:10.541210 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:10.541895 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:10.543241 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:10.543759 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:10.543897 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:10.547757 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:10.547762 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:10.547765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:10.547799 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:10.550391 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:10.551039 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:10.551046 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:10.551050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:10.551209 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:10.551653 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:10.551658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:10.551662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:10.551922 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:10.552181 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:10.552489 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 20:16:10.552714 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:16:10.554512 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 28504
I20260929 20:16:10.567795 28910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:10.555671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35232 (local address 127.27.185.65:41125)
0929 20:16:10.555877 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:10.555882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:10.556076 (+ 194us) server_negotiation.cc:407] Connection header received
0929 20:16:10.556259 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:10.556264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:10.556333 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:10.556432 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:10.557612 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:10.558369 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:10.559706 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:10.560143 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:10.560275 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:10.562534 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:10.562540 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:10.562545 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:10.562585 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:10.565081 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:10.565770 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:10.565775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:10.565778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:10.565929 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:10.566315 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:10.566323 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:10.566327 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:10.566644 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:10.566893 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:10.567179 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 20:16:10.567421 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:16:10.569316 28628 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:16:10.656602 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 28504
I20260929 20:16:10.674160 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 28639
Sep 29 20:16:10 dist-test-slave-4cx3 krb5kdc[28396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.66@KRBTEST.COM
I20260929 20:16:10.690834 28917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:10.675017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34856 (local address 127.27.185.66:44091)
0929 20:16:10.675626 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:10.675633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:10.675659 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:16:10.675722 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:10.675728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:10.675960 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:10.676104 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:10.677455 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:10.678533 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:10.679858 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:10.680442 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:10.680593 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:10.684878 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:10.684890 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:10.684905 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:10.684952 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:10.687885 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:10.688504 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:10.688511 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:10.688514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:10.688635 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:10.689351 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:10.689354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:10.689356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:10.689715 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:10.689943 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:10.690214 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 20:16:10.690461 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":306,"threads_started":1}
I20260929 20:16:10.692304 28764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:16:10.777479 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 28639
I20260929 20:16:10.800711 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 28774
Sep 29 20:16:10 dist-test-slave-4cx3 krb5kdc[28396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.67@KRBTEST.COM
I20260929 20:16:10.817782 28919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:10.802250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49122 (local address 127.27.185.67:46347)
0929 20:16:10.802821 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:10.802828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:10.802860 (+ 32us) server_negotiation.cc:407] Connection header received
0929 20:16:10.802941 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:10.802948 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:10.803164 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:10.803294 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:10.804569 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:10.805560 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:10.806860 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:10.807588 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:10.807684 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:10.812011 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:10.812023 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:10.812038 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:10.812073 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:10.814786 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:10.815629 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:10.815636 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:10.815639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:10.815765 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:10.816177 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:10.816181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:10.816183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:10.816619 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:10.817005 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:10.817194 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 20:16:10.817428 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":285,"threads_started":1}
I20260929 20:16:10.819162 28898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:16:10.905185 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 28774
I20260929 20:16:10.922178 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 28409
I20260929 20:16:10.923341 28472 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:16:11.031145 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 28409
2026-09-29T20:16:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4081 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:16:13 dist-test-slave-4cx3 krb5kdc[28924](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:16:13 dist-test-slave-4cx3 krb5kdc[28924](info): set up 2 sockets
Sep 29 20:16:13 dist-test-slave-4cx3 krb5kdc[28924](info): commencing operation
krb5kdc: starting...
W20260929 20:16:14.300004 28389 mini_kdc.cc:121] Time spent starting KDC: real 3.226s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260929 20:16:14.491633 28389 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.191s user 0.000s sys 0.006s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260929 20:16:14.685868 28389 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.194s user 0.000s sys 0.005s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/krb5kdc/test-user.keytab.
W20260929 20:16:15.510025 28389 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.746s user 0.001s sys 0.005s
Sep 29 20:16:15 dist-test-slave-4cx3 krb5kdc[28924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:16:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:16:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
W20260929 20:16:15.754670 28389 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.27.185.126: real 0.197s user 0.004s sys 0.004s
I20260929 20:16:15.755807 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:36479
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:40167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:36479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:16.075518 28938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:16.075894 28938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:16.076064 28938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:16.083451 28938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:16:16.083617 28938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:16.083726 28938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:16.083833 28938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:16:16.083922 28938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:16:16.094185 28938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:40167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:36479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:36479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:16.095638 28938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:16.096957 28938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:16.109122 28946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:16.109074 28943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:16.109146 28944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:16.109838 28938 server_base.cc:1034] running on GCE node
I20260929 20:16:16.110620 28938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:16.112183 28938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:16.113426 28938 hybrid_clock.cc:648] HybridClock initialized: now 1790712976113350 us; error 86 us; skew 500 ppm
I20260929 20:16:16.113829 28938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:16 dist-test-slave-4cx3 krb5kdc[28924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712976, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:16.116963 28938 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:16:16.118465 28938 webserver.cc:460] Webserver started at http://127.27.185.126:38745/ using document root <none> and password file <none>
I20260929 20:16:16.119194 28938 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:16.119349 28938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:16.119705 28938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:16.122735 28938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "a8b50e4c8e5d46c5aacba72012f60c4f"
format_stamp: "Formatted at 2026-09-29 20:16:16 on dist-test-slave-4cx3"
server_key: "e99ec602bd7bfff2fc91b4b79f0c0127"
server_key_iv: "27204dbdab4e9ef40cdd93aecddbee66"
server_key_version: "encryptionkey@0"
I20260929 20:16:16.123581 28938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "a8b50e4c8e5d46c5aacba72012f60c4f"
format_stamp: "Formatted at 2026-09-29 20:16:16 on dist-test-slave-4cx3"
server_key: "e99ec602bd7bfff2fc91b4b79f0c0127"
server_key_iv: "27204dbdab4e9ef40cdd93aecddbee66"
server_key_version: "encryptionkey@0"
I20260929 20:16:16.128931 28938 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 20:16:16.132541 28953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:16.133931 28938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 20:16:16.134140 28938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "a8b50e4c8e5d46c5aacba72012f60c4f"
format_stamp: "Formatted at 2026-09-29 20:16:16 on dist-test-slave-4cx3"
server_key: "e99ec602bd7bfff2fc91b4b79f0c0127"
server_key_iv: "27204dbdab4e9ef40cdd93aecddbee66"
server_key_version: "encryptionkey@0"
I20260929 20:16:16.134362 28938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:16.149683 28938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:16.153858 28938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:16.154158 28938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:16.172978 28938 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:36479
I20260929 20:16:16.172983 29004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:36479 every 8 connection(s)
I20260929 20:16:16.174417 28938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:16:16.178149 29005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:16.183358 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 28938
I20260929 20:16:16.183617 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:16:16.184290 28389 external_mini_cluster.cc:1428] Setting key c3b4ec289751d5d8d6bb9e9db5262b0d
I20260929 20:16:16.189652 29005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8b50e4c8e5d46c5aacba72012f60c4f: Bootstrap starting.
I20260929 20:16:16.193481 29005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8b50e4c8e5d46c5aacba72012f60c4f: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:16.194741 29005 log.cc:824] T 00000000000000000000000000000000 P a8b50e4c8e5d46c5aacba72012f60c4f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:16:16.198338 29005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8b50e4c8e5d46c5aacba72012f60c4f: No bootstrap required, opened a new log
Sep 29 20:16:16 dist-test-slave-4cx3 krb5kdc[28924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:16.207935 29005 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8b50e4c8e5d46c5aacba72012f60c4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b50e4c8e5d46c5aacba72012f60c4f" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36479 } }
I20260929 20:16:16.208448 29005 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8b50e4c8e5d46c5aacba72012f60c4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:16.208559 29005 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8b50e4c8e5d46c5aacba72012f60c4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8b50e4c8e5d46c5aacba72012f60c4f, State: Initialized, Role: FOLLOWER
I20260929 20:16:16.209304 29005 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8b50e4c8e5d46c5aacba72012f60c4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b50e4c8e5d46c5aacba72012f60c4f" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36479 } }
I20260929 20:16:16.209610 29005 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8b50e4c8e5d46c5aacba72012f60c4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:16:16.209755 29005 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8b50e4c8e5d46c5aacba72012f60c4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:16:16.209950 29008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:16.186325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37018 (local address 127.27.185.126:36479)
0929 20:16:16.187187 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:16.187199 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:16.187239 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:16:16.189020 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:16.189044 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:16.189472 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:16.189852 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:16.191783 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:16.193077 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:16.194908 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:16.195985 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:16.196156 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:16.200970 (+ 4814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:16.200978 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:16.200993 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:16.201042 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:16.205027 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:16.205729 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:16.205743 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:16.205754 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:16.205993 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:16.206358 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:16.206363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:16.206367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:16.207155 (+ 788us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:16.207471 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:16.207960 (+ 489us) server_negotiation.cc:299] Negotiation successful
0929 20:16:16.208444 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":425,"threads_started":1}
I20260929 20:16:16.209970 29005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8b50e4c8e5d46c5aacba72012f60c4f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:16.212558 29005 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8b50e4c8e5d46c5aacba72012f60c4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b50e4c8e5d46c5aacba72012f60c4f" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36479 } }
I20260929 20:16:16.213080 29005 leader_election.cc:302] T 00000000000000000000000000000000 P a8b50e4c8e5d46c5aacba72012f60c4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8b50e4c8e5d46c5aacba72012f60c4f; no voters:
I20260929 20:16:16.213666 29005 leader_election.cc:288] T 00000000000000000000000000000000 P a8b50e4c8e5d46c5aacba72012f60c4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:16:16.213804 29010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8b50e4c8e5d46c5aacba72012f60c4f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:16:16.214525 29010 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8b50e4c8e5d46c5aacba72012f60c4f [term 1 LEADER]: Becoming Leader. State: Replica: a8b50e4c8e5d46c5aacba72012f60c4f, State: Running, Role: LEADER
I20260929 20:16:16.215068 29010 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8b50e4c8e5d46c5aacba72012f60c4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b50e4c8e5d46c5aacba72012f60c4f" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36479 } }
I20260929 20:16:16.215888 29005 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8b50e4c8e5d46c5aacba72012f60c4f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:16:16.218212 29012 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8b50e4c8e5d46c5aacba72012f60c4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8b50e4c8e5d46c5aacba72012f60c4f. Latest consensus state: current_term: 1 leader_uuid: "a8b50e4c8e5d46c5aacba72012f60c4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b50e4c8e5d46c5aacba72012f60c4f" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36479 } } }
I20260929 20:16:16.218189 29011 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8b50e4c8e5d46c5aacba72012f60c4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8b50e4c8e5d46c5aacba72012f60c4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b50e4c8e5d46c5aacba72012f60c4f" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36479 } } }
I20260929 20:16:16.218681 29011 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8b50e4c8e5d46c5aacba72012f60c4f [sys.catalog]: This master's current role is: LEADER
I20260929 20:16:16.223341 29012 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8b50e4c8e5d46c5aacba72012f60c4f [sys.catalog]: This master's current role is: LEADER
I20260929 20:16:16.224387 29017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:16:16.228889 29017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:16:16.236943 29017 catalog_manager.cc:1347] Generated new cluster ID: fa4dde118a3440399889c5357e469cdd
I20260929 20:16:16.237056 29017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:16:16.256417 29017 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:16:16.257858 29017 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:16:16.266081 29017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8b50e4c8e5d46c5aacba72012f60c4f: Generated new TSK 0
I20260929 20:16:16.266942 29017 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
I20260929 20:16:16.315325 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:0
--local_ip_for_outbound_sockets=127.27.185.65
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:36479
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:40167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:16.648752 29031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:16.649115 29031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:16.649312 29031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:16.656862 29031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:16.657043 29031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:16.657236 29031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:16:16.667975 29031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:40167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:36479
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:16.669486 29031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:16.670940 29031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:16.684602 29039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:16.684674 29036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:16.684648 29037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:16.685668 29031 server_base.cc:1034] running on GCE node
I20260929 20:16:16.686268 29031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:16.687275 29031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:16.688519 29031 hybrid_clock.cc:648] HybridClock initialized: now 1790712976688461 us; error 40 us; skew 500 ppm
I20260929 20:16:16.688896 29031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:16 dist-test-slave-4cx3 krb5kdc[28924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712976, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:16.692369 29031 init.cc:377] Logged in from keytab as kudu/127.27.185.65@KRBTEST.COM (short username kudu)
I20260929 20:16:16.693959 29031 webserver.cc:460] Webserver started at http://127.27.185.65:43327/ using document root <none> and password file <none>
I20260929 20:16:16.694685 29031 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:16.694841 29031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:16.695169 29031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:16.698247 29031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-0/data/instance:
uuid: "e5dbafb6704b4a268fcfc7a7d63493e8"
format_stamp: "Formatted at 2026-09-29 20:16:16 on dist-test-slave-4cx3"
server_key: "99a1b3ee7849660f63c373a336d573e5"
server_key_iv: "c085a750bd6449841a25db3d09ddbb73"
server_key_version: "encryptionkey@0"
I20260929 20:16:16.699076 29031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance:
uuid: "e5dbafb6704b4a268fcfc7a7d63493e8"
format_stamp: "Formatted at 2026-09-29 20:16:16 on dist-test-slave-4cx3"
server_key: "99a1b3ee7849660f63c373a336d573e5"
server_key_iv: "c085a750bd6449841a25db3d09ddbb73"
server_key_version: "encryptionkey@0"
I20260929 20:16:16.704474 29031 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 20:16:16.708078 29046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:16.709740 29031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:16:16.709941 29031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "e5dbafb6704b4a268fcfc7a7d63493e8"
format_stamp: "Formatted at 2026-09-29 20:16:16 on dist-test-slave-4cx3"
server_key: "99a1b3ee7849660f63c373a336d573e5"
server_key_iv: "c085a750bd6449841a25db3d09ddbb73"
server_key_version: "encryptionkey@0"
I20260929 20:16:16.710150 29031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:16.726358 29031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:16.730700 29031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:16.731020 29031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:16.732125 29031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:16:16.733642 29031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:16:16.733745 29031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:16.733920 29031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:16:16.733989 29031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:16.767727 29031 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:39161
I20260929 20:16:16.767771 29159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:39161 every 8 connection(s)
I20260929 20:16:16.769066 29031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:16:16.774808 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 29031
I20260929 20:16:16.775018 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance
I20260929 20:16:16.775717 28389 external_mini_cluster.cc:1428] Setting key b38b99c452634c2549e959891cff59cf
Sep 29 20:16:16 dist-test-slave-4cx3 krb5kdc[28924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712976, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:16.789950 29162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:16.771815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:49851 (local address 127.27.185.126:36479)
0929 20:16:16.772423 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:16.772431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:16.773417 (+ 986us) server_negotiation.cc:407] Connection header received
0929 20:16:16.775317 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:16.775323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:16.775450 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:16.775573 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:16.777256 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:16.778165 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:16.779832 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:16.780225 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:16.780377 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:16.784231 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:16.784235 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:16.784239 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:16.784272 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:16.786788 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:16.787592 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:16.787601 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:16.787604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:16.787719 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:16.788183 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:16.788200 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:16.788202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:16.788463 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:16.788613 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:16.789343 (+ 730us) server_negotiation.cc:299] Negotiation successful
0929 20:16:16.789576 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":246,"threads_started":1}
I20260929 20:16:16.790979 29163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:16.772433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36479 (local address 127.27.185.65:49851)
0929 20:16:16.773200 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:16.773245 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:16.775011 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:16.775839 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:16.775852 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:16.776507 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:16.777088 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:16.777105 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:16.778335 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:16.778340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:16.779674 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:16.779686 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:16.780464 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:16.780473 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:16.780734 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:16.781477 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:16.781501 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:16.783991 (+ 2490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:16.786950 (+ 2959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:16.786959 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:16.786977 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:16.787457 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:16.787834 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:16.787840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:16.787844 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:16.788041 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:16.788597 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:16.788605 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:16.788960 (+ 355us) client_negotiation.cc:769] Sending connection context
0929 20:16:16.789220 (+ 260us) client_negotiation.cc:240] Negotiation successful
0929 20:16:16.789796 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":281,"threads_started":1}
WARNING: no policy specified for kudu/127.27.185.66@KRBTEST.COM; defaulting to no policy
I20260929 20:16:16.792695 29160 heartbeater.cc:347] Connected to a master server at 127.27.185.126:36479
I20260929 20:16:16.793082 29160 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.66@KRBTEST.COM" created.
I20260929 20:16:16.794034 29160 heartbeater.cc:511] Master 127.27.185.126:36479 requested a full tablet report, sending...
I20260929 20:16:16.796345 28970 ts_manager.cc:194] Registered new tserver with Master: e5dbafb6704b4a268fcfc7a7d63493e8 (127.27.185.65:39161)
I20260929 20:16:16.797999 28970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.65@KRBTEST.COM'} at 127.27.185.65:49851
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
I20260929 20:16:16.816738 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:0
--local_ip_for_outbound_sockets=127.27.185.66
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:36479
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:40167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:17.140069 29166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:17.140354 29166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:17.140452 29166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:17.147841 29166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:17.147953 29166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:17.148069 29166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:16:17.158499 29166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:40167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:36479
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:17.159934 29166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:17.161301 29166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:17.174866 29172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:17.174889 29174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:17.175403 29171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:17.181043 29166 server_base.cc:1034] running on GCE node
I20260929 20:16:17.181674 29166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:17.182687 29166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:17.183954 29166 hybrid_clock.cc:648] HybridClock initialized: now 1790712977183885 us; error 57 us; skew 500 ppm
I20260929 20:16:17.184326 29166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:17 dist-test-slave-4cx3 krb5kdc[28924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712977, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:17.187543 29166 init.cc:377] Logged in from keytab as kudu/127.27.185.66@KRBTEST.COM (short username kudu)
I20260929 20:16:17.189134 29166 webserver.cc:460] Webserver started at http://127.27.185.66:44645/ using document root <none> and password file <none>
I20260929 20:16:17.189851 29166 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:17.189980 29166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:17.190306 29166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:17.193406 29166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-1/data/instance:
uuid: "2f70be30937e4ba8ae3f357959d43c88"
format_stamp: "Formatted at 2026-09-29 20:16:17 on dist-test-slave-4cx3"
server_key: "2eb9d789094b4a81dc2b7fe722b3ecf2"
server_key_iv: "6eacc603a6a535e1c5b32c5f4be5767b"
server_key_version: "encryptionkey@0"
I20260929 20:16:17.194262 29166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance:
uuid: "2f70be30937e4ba8ae3f357959d43c88"
format_stamp: "Formatted at 2026-09-29 20:16:17 on dist-test-slave-4cx3"
server_key: "2eb9d789094b4a81dc2b7fe722b3ecf2"
server_key_iv: "6eacc603a6a535e1c5b32c5f4be5767b"
server_key_version: "encryptionkey@0"
I20260929 20:16:17.199378 29166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 20:16:17.202844 29181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:17.204205 29166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:16:17.204366 29166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "2f70be30937e4ba8ae3f357959d43c88"
format_stamp: "Formatted at 2026-09-29 20:16:17 on dist-test-slave-4cx3"
server_key: "2eb9d789094b4a81dc2b7fe722b3ecf2"
server_key_iv: "6eacc603a6a535e1c5b32c5f4be5767b"
server_key_version: "encryptionkey@0"
I20260929 20:16:17.204587 29166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:17.220628 29166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:17.224821 29166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:17.225119 29166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:17.226125 29166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:16:17.227524 29166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:16:17.227596 29166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:17.227738 29166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:16:17.227788 29166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:17.261493 29166 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:38533
I20260929 20:16:17.261538 29294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:38533 every 8 connection(s)
I20260929 20:16:17.262836 29166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:16:17.264911 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 29166
I20260929 20:16:17.265105 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance
I20260929 20:16:17.265614 28389 external_mini_cluster.cc:1428] Setting key 0493fda3236160abf60155cd0899c6d8
Sep 29 20:16:17 dist-test-slave-4cx3 krb5kdc[28924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712977, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.185.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.67@KRBTEST.COM" created.
I20260929 20:16:17.287627 29162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:17.265736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:53895 (local address 127.27.185.126:36479)
0929 20:16:17.265947 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:17.265953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:17.267299 (+ 1346us) server_negotiation.cc:407] Connection header received
0929 20:16:17.270264 (+ 2965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:17.270269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:17.270338 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:17.270427 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:17.272653 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:17.273537 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:17.275500 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:17.276301 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:17.276452 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:17.280404 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:17.280413 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:17.280422 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:17.280473 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:17.284231 (+ 3758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:17.285085 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:17.285089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:17.285091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:17.285216 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:17.285707 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:17.285710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:17.285712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:17.285969 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:17.286097 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:17.286852 (+ 755us) server_negotiation.cc:299] Negotiation successful
0929 20:16:17.287157 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:16:17.288693 29297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:17.266398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36479 (local address 127.27.185.66:53895)
0929 20:16:17.267146 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:17.267196 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:17.269956 (+ 2760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:17.270732 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:17.270746 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:17.271602 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:17.272458 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:17.272487 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:17.273713 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:17.273720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:17.275314 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:17.275327 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:17.276521 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:17.276527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:17.276729 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:17.277479 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:17.277504 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:17.280114 (+ 2610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:17.284400 (+ 4286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:17.284430 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:17.284445 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:17.284949 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:17.285312 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:17.285315 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:17.285317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:17.285580 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:17.286091 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:17.286097 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:17.286510 (+ 413us) client_negotiation.cc:769] Sending connection context
0929 20:16:17.286743 (+ 233us) client_negotiation.cc:240] Negotiation successful
0929 20:16:17.287203 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":329,"threads_started":1}
I20260929 20:16:17.290230 29295 heartbeater.cc:347] Connected to a master server at 127.27.185.126:36479
I20260929 20:16:17.290566 29295 heartbeater.cc:464] Registering TS with master...
I20260929 20:16:17.291309 29295 heartbeater.cc:511] Master 127.27.185.126:36479 requested a full tablet report, sending...
I20260929 20:16:17.292874 28970 ts_manager.cc:194] Registered new tserver with Master: 2f70be30937e4ba8ae3f357959d43c88 (127.27.185.66:38533)
I20260929 20:16:17.293695 28970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.66@KRBTEST.COM'} at 127.27.185.66:53895
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
I20260929 20:16:17.310905 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:0
--local_ip_for_outbound_sockets=127.27.185.67
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:36479
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:40167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:17.633653 29300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:17.633980 29300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:17.634160 29300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:17.641598 29300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:17.641747 29300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:17.641937 29300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:16:17.652261 29300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:40167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:36479
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:17.653671 29300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:17.655071 29300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:17.668531 29305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:17.668493 29308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:17.668468 29306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:17.669215 29300 server_base.cc:1034] running on GCE node
I20260929 20:16:17.669915 29300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:17.670886 29300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:17.672116 29300 hybrid_clock.cc:648] HybridClock initialized: now 1790712977672070 us; error 38 us; skew 500 ppm
I20260929 20:16:17.672516 29300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:17 dist-test-slave-4cx3 krb5kdc[28924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712977, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:17.680806 29300 init.cc:377] Logged in from keytab as kudu/127.27.185.67@KRBTEST.COM (short username kudu)
I20260929 20:16:17.682245 29300 webserver.cc:460] Webserver started at http://127.27.185.67:36071/ using document root <none> and password file <none>
I20260929 20:16:17.682941 29300 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:17.683090 29300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:17.683483 29300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:17.686422 29300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-2/data/instance:
uuid: "c74a63e7176547dead710f149763c3a4"
format_stamp: "Formatted at 2026-09-29 20:16:17 on dist-test-slave-4cx3"
server_key: "5b3d7bf3c0f87eca505ec1b839fddc15"
server_key_iv: "69a4387a027963c6de83993404a638b0"
server_key_version: "encryptionkey@0"
I20260929 20:16:17.687204 29300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance:
uuid: "c74a63e7176547dead710f149763c3a4"
format_stamp: "Formatted at 2026-09-29 20:16:17 on dist-test-slave-4cx3"
server_key: "5b3d7bf3c0f87eca505ec1b839fddc15"
server_key_iv: "69a4387a027963c6de83993404a638b0"
server_key_version: "encryptionkey@0"
I20260929 20:16:17.692366 29300 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 20:16:17.695827 29315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:17.697149 29300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 20:16:17.697327 29300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "c74a63e7176547dead710f149763c3a4"
format_stamp: "Formatted at 2026-09-29 20:16:17 on dist-test-slave-4cx3"
server_key: "5b3d7bf3c0f87eca505ec1b839fddc15"
server_key_iv: "69a4387a027963c6de83993404a638b0"
server_key_version: "encryptionkey@0"
I20260929 20:16:17.697551 29300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:17.721969 29300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:17.726384 29300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:17.726706 29300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:17.727730 29300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:16:17.729095 29300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:16:17.729197 29300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:17.729358 29300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:16:17.729434 29300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:17.763798 29300 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:37023
I20260929 20:16:17.763890 29429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:37023 every 8 connection(s)
I20260929 20:16:17.765245 29300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:16:17.767983 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 29300
I20260929 20:16:17.768150 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance
I20260929 20:16:17.768676 28389 external_mini_cluster.cc:1428] Setting key 711751d9ead254e07a74eb9213d7f63f
Sep 29 20:16:17 dist-test-slave-4cx3 krb5kdc[28924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712977, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:17.786942 29162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:17.767941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:42943 (local address 127.27.185.126:36479)
0929 20:16:17.768145 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:17.768153 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:17.769579 (+ 1426us) server_negotiation.cc:407] Connection header received
0929 20:16:17.772303 (+ 2724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:17.772307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:17.772374 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:17.772458 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:17.774237 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:17.774990 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:17.777080 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:17.777466 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:17.777563 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:17.780848 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:17.780852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:17.780856 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:17.780890 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:17.784125 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:17.784945 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:17.784948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:17.784951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:17.785066 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:17.785450 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:17.785452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:17.785454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:17.785711 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:17.785816 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:17.786535 (+ 719us) server_negotiation.cc:299] Negotiation successful
0929 20:16:17.786691 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:16:17.787967 29432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:17.768505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36479 (local address 127.27.185.67:42943)
0929 20:16:17.769393 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:17.769450 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:17.771976 (+ 2526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:17.772741 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:17.772753 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:17.773466 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:17.774088 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:17.774105 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:17.775119 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:17.775123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:17.776943 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:17.776954 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:17.777650 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:17.777654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:17.777852 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:17.778464 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:17.778485 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:17.780664 (+ 2179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:17.784273 (+ 3609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:17.784283 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:17.784300 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:17.784823 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:17.785157 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:17.785159 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:17.785161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:17.785331 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:17.785829 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:17.785835 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:17.786163 (+ 328us) client_negotiation.cc:769] Sending connection context
0929 20:16:17.786409 (+ 246us) client_negotiation.cc:240] Negotiation successful
0929 20:16:17.786797 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":342,"threads_started":1}
I20260929 20:16:17.789490 29430 heartbeater.cc:347] Connected to a master server at 127.27.185.126:36479
I20260929 20:16:17.789767 29430 heartbeater.cc:464] Registering TS with master...
I20260929 20:16:17.790474 29430 heartbeater.cc:511] Master 127.27.185.126:36479 requested a full tablet report, sending...
I20260929 20:16:17.791932 28970 ts_manager.cc:194] Registered new tserver with Master: c74a63e7176547dead710f149763c3a4 (127.27.185.67:37023)
I20260929 20:16:17.792747 28970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.67@KRBTEST.COM'} at 127.27.185.67:42943
I20260929 20:16:17.796897 28389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:16:17.801988 29160 heartbeater.cc:503] Master 127.27.185.126:36479 was elected leader, sending a full tablet report...
I20260929 20:16:17.816424 29162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:17.803463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37026 (local address 127.27.185.126:36479)
0929 20:16:17.803647 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:17.803654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:17.804233 (+ 579us) server_negotiation.cc:407] Connection header received
0929 20:16:17.804364 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:17.804368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:17.804437 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:17.804520 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:17.806090 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:17.806805 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:17.808118 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:17.808570 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:17.808670 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:17.810965 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:17.810970 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:17.810973 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:17.811016 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:17.813766 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:17.814575 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:17.814579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:17.814581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:17.814713 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:17.815062 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:17.815065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:17.815067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:17.815364 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:17.815604 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:17.815934 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 20:16:17.816121 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:16:17.823050 28970 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:16:17.829775 28970 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:16:17.856609 29446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:17.843105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55700 (local address 127.27.185.66:38533)
0929 20:16:17.843665 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:17.843674 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:17.847358 (+ 3684us) server_negotiation.cc:407] Connection header received
0929 20:16:17.847793 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:17.847807 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:17.848187 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:17.848381 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:16:17.849140 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:17.850815 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:17.853689 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:17.855039 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:17.855180 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:17.855471 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:17.856078 (+ 607us) server_negotiation.cc:299] Negotiation successful
0929 20:16:17.856263 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":1028,"threads_started":1}
I20260929 20:16:17.856765 29445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:17.842669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:37023 (local address 127.0.0.1:45044)
0929 20:16:17.848258 (+ 5589us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:17.848278 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:17.848412 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:17.849173 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:17.849179 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:17.849222 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:16:17.849727 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:17.849741 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:17.851874 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:17.851877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:17.853769 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:17.853783 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:17.855792 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:17.855796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:17.855947 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:17.855986 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 20:16:17.856081 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 20:16:17.856277 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5448,"mutex_wait_us":109,"spinlock_wait_cycles":1280,"thread_start_us":5317,"threads_started":1}
I20260929 20:16:17.857203 29444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:17.842524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44136 (local address 127.27.185.65:39161)
0929 20:16:17.848669 (+ 6145us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:17.848678 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:17.848702 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:16:17.848781 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:17.848787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:17.848965 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:17.849580 (+ 615us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:16:17.850098 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:17.851681 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:17.854374 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:17.855822 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:17.855984 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:17.856213 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:17.856372 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 20:16:17.856525 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5974,"thread_start_us":5790,"threads_started":1}
I20260929 20:16:17.857398 29447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:17.843801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45044 (local address 127.27.185.67:37023)
0929 20:16:17.847782 (+ 3981us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:17.847795 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:17.848552 (+ 757us) server_negotiation.cc:407] Connection header received
0929 20:16:17.848653 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:17.848661 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:17.848900 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:17.849121 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:16:17.849965 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:17.851675 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:17.853931 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:17.855348 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:17.855509 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:17.855674 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:17.856974 (+ 1300us) server_negotiation.cc:299] Negotiation successful
0929 20:16:17.857099 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3782,"thread_start_us":3622,"threads_started":1}
I20260929 20:16:17.856815 29442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:17.841969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:39161 (local address 127.0.0.1:44136)
0929 20:16:17.842749 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:17.842789 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:17.842971 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:17.849164 (+ 6193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:17.849170 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:17.849230 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:16:17.849852 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:17.849886 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:17.851833 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:17.851838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:17.854204 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:17.854218 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:17.856048 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:17.856056 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:17.856205 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:17.856233 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 20:16:17.856319 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 20:16:17.856442 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":25,"spinlock_wait_cycles":11904,"thread_start_us":946,"threads_started":1}
I20260929 20:16:17.857430 29443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:17.842114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:38533 (local address 127.0.0.1:55700)
0929 20:16:17.847234 (+ 5120us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:17.847256 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:17.847518 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:17.848678 (+ 1160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:17.848683 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:17.848707 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:16:17.848963 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:17.848971 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:17.850978 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:17.850982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:17.853511 (+ 2529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:17.853527 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:17.855597 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:17.855603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:17.855803 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:17.855860 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 20:16:17.856902 (+ 1042us) client_negotiation.cc:240] Negotiation successful
0929 20:16:17.857034 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4992,"mutex_wait_us":59,"thread_start_us":4828,"threads_started":1}
I20260929 20:16:17.861129 29095 tablet_service.cc:1467] Processing CreateTablet for tablet 1cef86fb7f1443debf9b311ec505c9e2 (DEFAULT_TABLE table=test-table [id=d4e0f892e8b843b78ff728d5461b9bdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:16:17.861425 29363 tablet_service.cc:1467] Processing CreateTablet for tablet 1cef86fb7f1443debf9b311ec505c9e2 (DEFAULT_TABLE table=test-table [id=d4e0f892e8b843b78ff728d5461b9bdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:16:17.861588 29230 tablet_service.cc:1467] Processing CreateTablet for tablet 1cef86fb7f1443debf9b311ec505c9e2 (DEFAULT_TABLE table=test-table [id=d4e0f892e8b843b78ff728d5461b9bdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:16:17.863050 29230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cef86fb7f1443debf9b311ec505c9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:17.863050 29363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cef86fb7f1443debf9b311ec505c9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:17.863106 29095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cef86fb7f1443debf9b311ec505c9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:17.874935 29448 tablet_bootstrap.cc:492] T 1cef86fb7f1443debf9b311ec505c9e2 P e5dbafb6704b4a268fcfc7a7d63493e8: Bootstrap starting.
I20260929 20:16:17.877951 29449 tablet_bootstrap.cc:492] T 1cef86fb7f1443debf9b311ec505c9e2 P 2f70be30937e4ba8ae3f357959d43c88: Bootstrap starting.
I20260929 20:16:17.880126 29450 tablet_bootstrap.cc:492] T 1cef86fb7f1443debf9b311ec505c9e2 P c74a63e7176547dead710f149763c3a4: Bootstrap starting.
I20260929 20:16:17.880584 29448 tablet_bootstrap.cc:654] T 1cef86fb7f1443debf9b311ec505c9e2 P e5dbafb6704b4a268fcfc7a7d63493e8: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:17.881398 29449 tablet_bootstrap.cc:654] T 1cef86fb7f1443debf9b311ec505c9e2 P 2f70be30937e4ba8ae3f357959d43c88: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:17.881953 29448 log.cc:824] T 1cef86fb7f1443debf9b311ec505c9e2 P e5dbafb6704b4a268fcfc7a7d63493e8: Log is configured to *not* fsync() on all Append() calls
I20260929 20:16:17.882643 29449 log.cc:824] T 1cef86fb7f1443debf9b311ec505c9e2 P 2f70be30937e4ba8ae3f357959d43c88: Log is configured to *not* fsync() on all Append() calls
I20260929 20:16:17.883319 29450 tablet_bootstrap.cc:654] T 1cef86fb7f1443debf9b311ec505c9e2 P c74a63e7176547dead710f149763c3a4: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:17.884526 29450 log.cc:824] T 1cef86fb7f1443debf9b311ec505c9e2 P c74a63e7176547dead710f149763c3a4: Log is configured to *not* fsync() on all Append() calls
I20260929 20:16:17.885296 29449 tablet_bootstrap.cc:492] T 1cef86fb7f1443debf9b311ec505c9e2 P 2f70be30937e4ba8ae3f357959d43c88: No bootstrap required, opened a new log
I20260929 20:16:17.885298 29448 tablet_bootstrap.cc:492] T 1cef86fb7f1443debf9b311ec505c9e2 P e5dbafb6704b4a268fcfc7a7d63493e8: No bootstrap required, opened a new log
I20260929 20:16:17.885596 29448 ts_tablet_manager.cc:1397] T 1cef86fb7f1443debf9b311ec505c9e2 P e5dbafb6704b4a268fcfc7a7d63493e8: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260929 20:16:17.885617 29449 ts_tablet_manager.cc:1397] T 1cef86fb7f1443debf9b311ec505c9e2 P 2f70be30937e4ba8ae3f357959d43c88: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260929 20:16:17.887835 29450 tablet_bootstrap.cc:492] T 1cef86fb7f1443debf9b311ec505c9e2 P c74a63e7176547dead710f149763c3a4: No bootstrap required, opened a new log
I20260929 20:16:17.888108 29450 ts_tablet_manager.cc:1397] T 1cef86fb7f1443debf9b311ec505c9e2 P c74a63e7176547dead710f149763c3a4: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260929 20:16:17.892954 29448 raft_consensus.cc:348] T 1cef86fb7f1443debf9b311ec505c9e2 P e5dbafb6704b4a268fcfc7a7d63493e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5dbafb6704b4a268fcfc7a7d63493e8" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 39161 } } peers { permanent_uuid: "2f70be30937e4ba8ae3f357959d43c88" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38533 } } peers { permanent_uuid: "c74a63e7176547dead710f149763c3a4" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 37023 } }
I20260929 20:16:17.893344 29448 raft_consensus.cc:374] T 1cef86fb7f1443debf9b311ec505c9e2 P e5dbafb6704b4a268fcfc7a7d63493e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:17.893463 29448 raft_consensus.cc:729] T 1cef86fb7f1443debf9b311ec505c9e2 P e5dbafb6704b4a268fcfc7a7d63493e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5dbafb6704b4a268fcfc7a7d63493e8, State: Initialized, Role: FOLLOWER
I20260929 20:16:17.894119 29448 consensus_queue.cc:260] T 1cef86fb7f1443debf9b311ec505c9e2 P e5dbafb6704b4a268fcfc7a7d63493e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5dbafb6704b4a268fcfc7a7d63493e8" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 39161 } } peers { permanent_uuid: "2f70be30937e4ba8ae3f357959d43c88" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38533 } } peers { permanent_uuid: "c74a63e7176547dead710f149763c3a4" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 37023 } }
I20260929 20:16:17.895921 29448 ts_tablet_manager.cc:1428] T 1cef86fb7f1443debf9b311ec505c9e2 P e5dbafb6704b4a268fcfc7a7d63493e8: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260929 20:16:17.896130 29449 raft_consensus.cc:348] T 1cef86fb7f1443debf9b311ec505c9e2 P 2f70be30937e4ba8ae3f357959d43c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5dbafb6704b4a268fcfc7a7d63493e8" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 39161 } } peers { permanent_uuid: "2f70be30937e4ba8ae3f357959d43c88" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38533 } } peers { permanent_uuid: "c74a63e7176547dead710f149763c3a4" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 37023 } }
I20260929 20:16:17.896598 29449 raft_consensus.cc:374] T 1cef86fb7f1443debf9b311ec505c9e2 P 2f70be30937e4ba8ae3f357959d43c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:17.896699 29449 raft_consensus.cc:729] T 1cef86fb7f1443debf9b311ec505c9e2 P 2f70be30937e4ba8ae3f357959d43c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f70be30937e4ba8ae3f357959d43c88, State: Initialized, Role: FOLLOWER
I20260929 20:16:17.897405 29449 consensus_queue.cc:260] T 1cef86fb7f1443debf9b311ec505c9e2 P 2f70be30937e4ba8ae3f357959d43c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5dbafb6704b4a268fcfc7a7d63493e8" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 39161 } } peers { permanent_uuid: "2f70be30937e4ba8ae3f357959d43c88" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38533 } } peers { permanent_uuid: "c74a63e7176547dead710f149763c3a4" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 37023 } }
I20260929 20:16:17.898900 29295 heartbeater.cc:503] Master 127.27.185.126:36479 was elected leader, sending a full tablet report...
I20260929 20:16:17.899587 29449 ts_tablet_manager.cc:1428] T 1cef86fb7f1443debf9b311ec505c9e2 P 2f70be30937e4ba8ae3f357959d43c88: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260929 20:16:17.899523 29450 raft_consensus.cc:348] T 1cef86fb7f1443debf9b311ec505c9e2 P c74a63e7176547dead710f149763c3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5dbafb6704b4a268fcfc7a7d63493e8" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 39161 } } peers { permanent_uuid: "2f70be30937e4ba8ae3f357959d43c88" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38533 } } peers { permanent_uuid: "c74a63e7176547dead710f149763c3a4" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 37023 } }
I20260929 20:16:17.900045 29450 raft_consensus.cc:374] T 1cef86fb7f1443debf9b311ec505c9e2 P c74a63e7176547dead710f149763c3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:17.900154 29450 raft_consensus.cc:729] T 1cef86fb7f1443debf9b311ec505c9e2 P c74a63e7176547dead710f149763c3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c74a63e7176547dead710f149763c3a4, State: Initialized, Role: FOLLOWER
I20260929 20:16:17.900774 29450 consensus_queue.cc:260] T 1cef86fb7f1443debf9b311ec505c9e2 P c74a63e7176547dead710f149763c3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5dbafb6704b4a268fcfc7a7d63493e8" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 39161 } } peers { permanent_uuid: "2f70be30937e4ba8ae3f357959d43c88" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38533 } } peers { permanent_uuid: "c74a63e7176547dead710f149763c3a4" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 37023 } }
I20260929 20:16:17.902005 29430 heartbeater.cc:503] Master 127.27.185.126:36479 was elected leader, sending a full tablet report...
I20260929 20:16:17.902559 29450 ts_tablet_manager.cc:1428] T 1cef86fb7f1443debf9b311ec505c9e2 P c74a63e7176547dead710f149763c3a4: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260929 20:16:17.993327 29456 raft_consensus.cc:482] T 1cef86fb7f1443debf9b311ec505c9e2 P c74a63e7176547dead710f149763c3a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:16:17.993731 29456 raft_consensus.cc:504] T 1cef86fb7f1443debf9b311ec505c9e2 P c74a63e7176547dead710f149763c3a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5dbafb6704b4a268fcfc7a7d63493e8" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 39161 } } peers { permanent_uuid: "2f70be30937e4ba8ae3f357959d43c88" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38533 } } peers { permanent_uuid: "c74a63e7176547dead710f149763c3a4" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 37023 } }
I20260929 20:16:17.995702 29456 leader_election.cc:288] T 1cef86fb7f1443debf9b311ec505c9e2 P c74a63e7176547dead710f149763c3a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5dbafb6704b4a268fcfc7a7d63493e8 (127.27.185.65:39161), 2f70be30937e4ba8ae3f357959d43c88 (127.27.185.66:38533)
I20260929 20:16:18.002547 29446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:17.996200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:45911 (local address 127.27.185.66:38533)
0929 20:16:17.996426 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:17.996433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:17.996553 (+ 120us) server_negotiation.cc:407] Connection header received
0929 20:16:17.996761 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:17.996765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:17.996828 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:17.996965 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:16:17.997404 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:17.998409 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:18.000745 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:18.001745 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:18.001861 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:18.001913 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:18.002248 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 20:16:18.002341 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 20:16:18.002655 29432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:17.996266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:38533 (local address 127.27.185.67:45911)
0929 20:16:17.996446 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:17.996466 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:17.996598 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:17.996989 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:17.996994 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:17.997021 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:16:17.997244 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:17.997255 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:17.998555 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:17.998559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:18.000564 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:18.000579 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:18.001885 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:18.001893 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:18.002084 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:18.002123 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 20:16:18.002224 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 20:16:18.002365 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":1}
I20260929 20:16:18.003644 29457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:17.996267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:39161 (local address 127.27.185.67:55087)
0929 20:16:17.997430 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:17.997444 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:17.997541 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:17.997834 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:17.997836 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:17.997855 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:16:17.998120 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:17.998127 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:17.999257 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:17.999263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:18.001288 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:18.001303 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:18.002885 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:18.002891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:18.003074 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:18.003104 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 20:16:18.003206 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 20:16:18.003310 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1054,"mutex_wait_us":454,"thread_start_us":410,"threads_started":1}
I20260929 20:16:18.003671 29444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:17.996205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:55087 (local address 127.27.185.65:39161)
0929 20:16:17.996426 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:17.996433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:17.997552 (+ 1119us) server_negotiation.cc:407] Connection header received
0929 20:16:17.997634 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:17.997640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:17.997722 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:17.997815 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:16:17.998283 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:17.999103 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:18.001514 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:18.002686 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:18.002876 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:18.002962 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:18.003264 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 20:16:18.003432 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 20:16:18.003829 29250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cef86fb7f1443debf9b311ec505c9e2" candidate_uuid: "c74a63e7176547dead710f149763c3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f70be30937e4ba8ae3f357959d43c88" is_pre_election: true
I20260929 20:16:18.004421 29250 raft_consensus.cc:2454] T 1cef86fb7f1443debf9b311ec505c9e2 P 2f70be30937e4ba8ae3f357959d43c88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c74a63e7176547dead710f149763c3a4 in term 0.
I20260929 20:16:18.004690 29115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cef86fb7f1443debf9b311ec505c9e2" candidate_uuid: "c74a63e7176547dead710f149763c3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5dbafb6704b4a268fcfc7a7d63493e8" is_pre_election: true
I20260929 20:16:18.005100 29319 leader_election.cc:302] T 1cef86fb7f1443debf9b311ec505c9e2 P c74a63e7176547dead710f149763c3a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f70be30937e4ba8ae3f357959d43c88, c74a63e7176547dead710f149763c3a4; no voters:
I20260929 20:16:18.005177 29115 raft_consensus.cc:2454] T 1cef86fb7f1443debf9b311ec505c9e2 P e5dbafb6704b4a268fcfc7a7d63493e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c74a63e7176547dead710f149763c3a4 in term 0.
I20260929 20:16:18.005494 29456 raft_consensus.cc:2781] T 1cef86fb7f1443debf9b311ec505c9e2 P c74a63e7176547dead710f149763c3a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:16:18.005656 29456 raft_consensus.cc:482] T 1cef86fb7f1443debf9b311ec505c9e2 P c74a63e7176547dead710f149763c3a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:16:18.005786 29456 raft_consensus.cc:3037] T 1cef86fb7f1443debf9b311ec505c9e2 P c74a63e7176547dead710f149763c3a4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:18.008185 29456 raft_consensus.cc:504] T 1cef86fb7f1443debf9b311ec505c9e2 P c74a63e7176547dead710f149763c3a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5dbafb6704b4a268fcfc7a7d63493e8" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 39161 } } peers { permanent_uuid: "2f70be30937e4ba8ae3f357959d43c88" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38533 } } peers { permanent_uuid: "c74a63e7176547dead710f149763c3a4" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 37023 } }
I20260929 20:16:18.008781 29456 leader_election.cc:288] T 1cef86fb7f1443debf9b311ec505c9e2 P c74a63e7176547dead710f149763c3a4 [CANDIDATE]: Term 1 election: Requested vote from peers e5dbafb6704b4a268fcfc7a7d63493e8 (127.27.185.65:39161), 2f70be30937e4ba8ae3f357959d43c88 (127.27.185.66:38533)
I20260929 20:16:18.009255 29115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cef86fb7f1443debf9b311ec505c9e2" candidate_uuid: "c74a63e7176547dead710f149763c3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5dbafb6704b4a268fcfc7a7d63493e8"
I20260929 20:16:18.009256 29250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cef86fb7f1443debf9b311ec505c9e2" candidate_uuid: "c74a63e7176547dead710f149763c3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f70be30937e4ba8ae3f357959d43c88"
I20260929 20:16:18.009483 29250 raft_consensus.cc:3037] T 1cef86fb7f1443debf9b311ec505c9e2 P 2f70be30937e4ba8ae3f357959d43c88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:18.009472 29115 raft_consensus.cc:3037] T 1cef86fb7f1443debf9b311ec505c9e2 P e5dbafb6704b4a268fcfc7a7d63493e8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:18.012723 29250 raft_consensus.cc:2454] T 1cef86fb7f1443debf9b311ec505c9e2 P 2f70be30937e4ba8ae3f357959d43c88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c74a63e7176547dead710f149763c3a4 in term 1.
I20260929 20:16:18.012722 29115 raft_consensus.cc:2454] T 1cef86fb7f1443debf9b311ec505c9e2 P e5dbafb6704b4a268fcfc7a7d63493e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c74a63e7176547dead710f149763c3a4 in term 1.
I20260929 20:16:18.013123 29319 leader_election.cc:302] T 1cef86fb7f1443debf9b311ec505c9e2 P c74a63e7176547dead710f149763c3a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f70be30937e4ba8ae3f357959d43c88, c74a63e7176547dead710f149763c3a4; no voters:
I20260929 20:16:18.013379 29456 raft_consensus.cc:2781] T 1cef86fb7f1443debf9b311ec505c9e2 P c74a63e7176547dead710f149763c3a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:16:18.013800 29456 raft_consensus.cc:686] T 1cef86fb7f1443debf9b311ec505c9e2 P c74a63e7176547dead710f149763c3a4 [term 1 LEADER]: Becoming Leader. State: Replica: c74a63e7176547dead710f149763c3a4, State: Running, Role: LEADER
I20260929 20:16:18.014428 29456 consensus_queue.cc:237] T 1cef86fb7f1443debf9b311ec505c9e2 P c74a63e7176547dead710f149763c3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5dbafb6704b4a268fcfc7a7d63493e8" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 39161 } } peers { permanent_uuid: "2f70be30937e4ba8ae3f357959d43c88" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38533 } } peers { permanent_uuid: "c74a63e7176547dead710f149763c3a4" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 37023 } }
W20260929 20:16:18.014871 29296 tablet.cc:2369] T 1cef86fb7f1443debf9b311ec505c9e2 P 2f70be30937e4ba8ae3f357959d43c88: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:16:18.016886 29431 tablet.cc:2369] T 1cef86fb7f1443debf9b311ec505c9e2 P c74a63e7176547dead710f149763c3a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:16:18.019833 28970 catalog_manager.cc:5495] T 1cef86fb7f1443debf9b311ec505c9e2 P c74a63e7176547dead710f149763c3a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to c74a63e7176547dead710f149763c3a4 (127.27.185.67). New cstate: current_term: 1 leader_uuid: "c74a63e7176547dead710f149763c3a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5dbafb6704b4a268fcfc7a7d63493e8" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 39161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f70be30937e4ba8ae3f357959d43c88" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c74a63e7176547dead710f149763c3a4" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 37023 } health_report { overall_health: HEALTHY } } }
W20260929 20:16:18.021379 29161 tablet.cc:2369] T 1cef86fb7f1443debf9b311ec505c9e2 P e5dbafb6704b4a268fcfc7a7d63493e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 20:16:18 dist-test-slave-4cx3 krb5kdc[28924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.65@KRBTEST.COM
I20260929 20:16:18.055745 29444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:18.040635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44146 (local address 127.27.185.65:39161)
0929 20:16:18.040841 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:18.040850 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:18.041306 (+ 456us) server_negotiation.cc:407] Connection header received
0929 20:16:18.041515 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:18.041520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:18.041588 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:18.041708 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:18.043236 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:18.044039 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:18.045315 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:18.045754 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:18.045894 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:18.050028 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:18.050042 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:18.050058 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:18.050092 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:18.052927 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:18.053619 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:18.053628 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:18.053630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:18.053760 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:18.054153 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:18.054159 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:18.054162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:18.054534 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:18.054850 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:18.055135 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 20:16:18.055352 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Sep 29 20:16:18 dist-test-slave-4cx3 krb5kdc[28924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712978, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 20:16:18 dist-test-slave-4cx3 krb5kdc[28924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712978, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.185.65@KRBTEST.COM
I20260929 20:16:18.101751 29444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:18.087692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44158 (local address 127.27.185.65:39161)
0929 20:16:18.087934 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:18.087939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:18.088065 (+ 126us) server_negotiation.cc:407] Connection header received
0929 20:16:18.088219 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:18.088223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:18.088288 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:18.088375 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:18.089674 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:18.090397 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:18.091888 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:18.092286 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:18.092382 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:18.096494 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:18.096499 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:18.096510 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:18.096545 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:18.099118 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:18.099923 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:18.099927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:18.099929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:18.100059 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:18.100411 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:18.100414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:18.100417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:18.100682 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:18.100921 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:18.101228 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 20:16:18.101397 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
Sep 29 20:16:18 dist-test-slave-4cx3 krb5kdc[28924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 20:16:18 dist-test-slave-4cx3 krb5kdc[28924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.65@KRBTEST.COM
I20260929 20:16:18.147593 29444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:18.133834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44174 (local address 127.27.185.65:39161)
0929 20:16:18.134056 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:18.134064 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:18.134373 (+ 309us) server_negotiation.cc:407] Connection header received
0929 20:16:18.134606 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:18.134611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:18.134677 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:18.134787 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:18.136209 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:18.136902 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:18.138212 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:18.138663 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:18.138799 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:18.142588 (+ 3789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:18.142593 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:18.142597 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:18.142630 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:18.144987 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:18.145624 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:18.145631 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:18.145634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:18.145758 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:18.146122 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:18.146127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:18.146131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:18.146425 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:18.146686 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:18.146985 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 20:16:18.147228 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 20:16:18.149327 29074 tablet_service.cc:2908] Rejecting scan request for tablet 1cef86fb7f1443debf9b311ec505c9e2: Uninitialized: clean time has not yet been initialized
I20260929 20:16:18.152281 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 29031
I20260929 20:16:18.164721 29444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:18.153193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44184 (local address 127.27.185.65:39161)
0929 20:16:18.153358 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:18.153363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:18.153731 (+ 368us) server_negotiation.cc:407] Connection header received
0929 20:16:18.153849 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:18.153853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:18.153922 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:18.154002 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:18.155069 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:18.155788 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:18.157000 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:18.157425 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:18.157539 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:18.159810 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:18.159817 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:18.159828 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:18.159860 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:18.162452 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:18.163094 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:18.163098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:18.163100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:18.163229 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:18.163605 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:18.163608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:18.163610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:18.163890 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:18.164102 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:18.164306 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 20:16:18.164479 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 20:16:18.165374 29155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:16:18.256332 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 29031
I20260929 20:16:18.277299 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 29166
Sep 29 20:16:18 dist-test-slave-4cx3 krb5kdc[28924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.66@KRBTEST.COM
I20260929 20:16:18.291982 29446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:18.278162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55702 (local address 127.27.185.66:38533)
0929 20:16:18.278339 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:18.278344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:18.278366 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:16:18.278427 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:18.278433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:18.278510 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:18.278611 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:18.279805 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:18.280977 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:18.282239 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:18.282877 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:18.283003 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:18.286960 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:18.286972 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:18.286984 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:18.287020 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:18.289624 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:18.290207 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:18.290213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:18.290216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:18.290333 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:18.290655 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:18.290658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:18.290660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:18.291007 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:18.291240 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:18.291500 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 20:16:18.291704 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:16:18.292661 29290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:16:18.381664 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 29166
W20260929 20:16:18.398486 29319 connection.cc:535] client connection to 127.27.185.66:38533 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 20:16:18.398784 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 29300
W20260929 20:16:18.398790 29319 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 20:16:18.400451 29457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:18.399959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:38533 (local address 127.27.185.67:33515)
0929 20:16:18.400121 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:18.400263 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.66:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 20:16:18.400689 29319 consensus_peers.cc:487] T 1cef86fb7f1443debf9b311ec505c9e2 P c74a63e7176547dead710f149763c3a4 -> Peer 2f70be30937e4ba8ae3f357959d43c88 (127.27.185.66:38533): Couldn't send request to peer 2f70be30937e4ba8ae3f357959d43c88. Status: Network error: Client connection negotiation failed: client connection to 127.27.185.66:38533: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Sep 29 20:16:18 dist-test-slave-4cx3 krb5kdc[28924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.67@KRBTEST.COM
I20260929 20:16:18.413961 29474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:18.399575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45058 (local address 127.27.185.67:37023)
0929 20:16:18.400400 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:18.400406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:18.400424 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:16:18.400493 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:18.400500 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:18.400583 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:18.400702 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:18.402085 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:18.402876 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:18.404231 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:18.404624 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:18.404722 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:18.408743 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:18.408753 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:18.408768 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:18.408799 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:18.411519 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:18.412158 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:18.412164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:18.412166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:18.412298 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:18.412628 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:18.412631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:18.412633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:18.412937 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:18.413154 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:18.413396 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 20:16:18.413559 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":655,"threads_started":1}
I20260929 20:16:18.414613 29425 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:16:18.503831 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 29300
I20260929 20:16:18.525179 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 28938
I20260929 20:16:18.526369 29000 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:16:18.633837 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 28938
2026-09-29T20:16:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (7603 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:16:18 dist-test-slave-4cx3 krb5kdc[29479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:16:18 dist-test-slave-4cx3 krb5kdc[29479](info): set up 2 sockets
Sep 29 20:16:18 dist-test-slave-4cx3 krb5kdc[29479](info): commencing operation
krb5kdc: starting...
W20260929 20:16:19.722746 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:16:19 dist-test-slave-4cx3 krb5kdc[29479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:16:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:16:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
I20260929 20:16:19.911111 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:39527
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:39843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:39527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:20.230779 29493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:20.231130 29493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:20.231304 29493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:20.238634 29493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:16:20.238785 29493 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 20:16:20.238886 29493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:20.239007 29493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:20.239094 29493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:16:20.239187 29493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:16:20.249459 29493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:39843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:39527
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:39527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:20.250820 29493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:20.252219 29493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:20.264097 29498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:20.264096 29501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:20.264122 29499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:20.265555 29493 server_base.cc:1034] running on GCE node
I20260929 20:16:20.266274 29493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:20.267782 29493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:20.269040 29493 hybrid_clock.cc:648] HybridClock initialized: now 1790712980268967 us; error 41 us; skew 500 ppm
I20260929 20:16:20.269356 29493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:20 dist-test-slave-4cx3 krb5kdc[29479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712980, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:20.272682 29493 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:16:20.274121 29493 webserver.cc:460] Webserver started at http://127.27.185.126:38753/ using document root <none> and password file <none>
I20260929 20:16:20.274832 29493 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:20.274972 29493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:20.275323 29493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:20.278270 29493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "2739ae47e0fd4fa49075618456af760d"
format_stamp: "Formatted at 2026-09-29 20:16:20 on dist-test-slave-4cx3"
server_key: "b26afcc3759840ae3a208b29d3ccbfc3"
server_key_iv: "b871c8e8524c4e937767a48cd85de97c"
server_key_version: "encryptionkey@0"
I20260929 20:16:20.279049 29493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "2739ae47e0fd4fa49075618456af760d"
format_stamp: "Formatted at 2026-09-29 20:16:20 on dist-test-slave-4cx3"
server_key: "b26afcc3759840ae3a208b29d3ccbfc3"
server_key_iv: "b871c8e8524c4e937767a48cd85de97c"
server_key_version: "encryptionkey@0"
I20260929 20:16:20.284180 29493 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:16:20.287576 29508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:20.288813 29493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:16:20.289007 29493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "2739ae47e0fd4fa49075618456af760d"
format_stamp: "Formatted at 2026-09-29 20:16:20 on dist-test-slave-4cx3"
server_key: "b26afcc3759840ae3a208b29d3ccbfc3"
server_key_iv: "b871c8e8524c4e937767a48cd85de97c"
server_key_version: "encryptionkey@0"
I20260929 20:16:20.289218 29493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:20.301079 29493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:20.304764 29493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:20.305029 29493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:20.325995 29493 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:39527
I20260929 20:16:20.326018 29569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:39527 every 8 connection(s)
I20260929 20:16:20.327366 29493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:16:20.328032 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 29493
I20260929 20:16:20.328248 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:16:20.329110 28389 external_mini_cluster.cc:1428] Setting key 9840d6e95fb26a84100aa103f9e695e9
I20260929 20:16:20.331909 29570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 20:16:20 dist-test-slave-4cx3 krb5kdc[29479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:20.342208 29570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2739ae47e0fd4fa49075618456af760d: Bootstrap starting.
I20260929 20:16:20.345718 29570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2739ae47e0fd4fa49075618456af760d: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:20.346766 29570 log.cc:824] T 00000000000000000000000000000000 P 2739ae47e0fd4fa49075618456af760d: Log is configured to *not* fsync() on all Append() calls
I20260929 20:16:20.349704 29570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2739ae47e0fd4fa49075618456af760d: No bootstrap required, opened a new log
I20260929 20:16:20.350772 29574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:20.331071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48230 (local address 127.27.185.126:39527)
0929 20:16:20.331858 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:20.331871 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:20.331913 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:16:20.333546 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:20.333565 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:20.333922 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:20.334329 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:20.335831 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:20.337169 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:20.338375 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:20.339044 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:20.339249 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:20.343263 (+ 4014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:20.343272 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:20.343286 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:20.343322 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:20.346517 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:20.347288 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:20.347295 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:20.347301 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:20.347547 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:20.347942 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:20.347947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:20.347951 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:20.348515 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:20.348807 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:20.349287 (+ 480us) server_negotiation.cc:299] Negotiation successful
0929 20:16:20.349738 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":397,"threads_started":1}
I20260929 20:16:20.358580 29570 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2739ae47e0fd4fa49075618456af760d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2739ae47e0fd4fa49075618456af760d" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 39527 } }
I20260929 20:16:20.358955 29570 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2739ae47e0fd4fa49075618456af760d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:20.359073 29570 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2739ae47e0fd4fa49075618456af760d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2739ae47e0fd4fa49075618456af760d, State: Initialized, Role: FOLLOWER
I20260929 20:16:20.359772 29570 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2739ae47e0fd4fa49075618456af760d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2739ae47e0fd4fa49075618456af760d" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 39527 } }
I20260929 20:16:20.360014 29570 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2739ae47e0fd4fa49075618456af760d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:16:20.360167 29570 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2739ae47e0fd4fa49075618456af760d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:16:20.360352 29570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2739ae47e0fd4fa49075618456af760d [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:20.362773 29570 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2739ae47e0fd4fa49075618456af760d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2739ae47e0fd4fa49075618456af760d" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 39527 } }
I20260929 20:16:20.363273 29570 leader_election.cc:302] T 00000000000000000000000000000000 P 2739ae47e0fd4fa49075618456af760d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2739ae47e0fd4fa49075618456af760d; no voters:
I20260929 20:16:20.363855 29570 leader_election.cc:288] T 00000000000000000000000000000000 P 2739ae47e0fd4fa49075618456af760d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:16:20.363970 29576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2739ae47e0fd4fa49075618456af760d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:16:20.364754 29576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2739ae47e0fd4fa49075618456af760d [term 1 LEADER]: Becoming Leader. State: Replica: 2739ae47e0fd4fa49075618456af760d, State: Running, Role: LEADER
I20260929 20:16:20.365298 29576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2739ae47e0fd4fa49075618456af760d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2739ae47e0fd4fa49075618456af760d" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 39527 } }
I20260929 20:16:20.366003 29570 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2739ae47e0fd4fa49075618456af760d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:16:20.367868 29578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2739ae47e0fd4fa49075618456af760d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2739ae47e0fd4fa49075618456af760d. Latest consensus state: current_term: 1 leader_uuid: "2739ae47e0fd4fa49075618456af760d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2739ae47e0fd4fa49075618456af760d" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 39527 } } }
I20260929 20:16:20.368266 29578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2739ae47e0fd4fa49075618456af760d [sys.catalog]: This master's current role is: LEADER
I20260929 20:16:20.368269 29577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2739ae47e0fd4fa49075618456af760d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2739ae47e0fd4fa49075618456af760d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2739ae47e0fd4fa49075618456af760d" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 39527 } } }
I20260929 20:16:20.370030 29577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2739ae47e0fd4fa49075618456af760d [sys.catalog]: This master's current role is: LEADER
I20260929 20:16:20.371551 29582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:16:20.376178 29582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:16:20.384725 29582 catalog_manager.cc:1347] Generated new cluster ID: 0e0159788d344dff9ecc150a6fa56930
I20260929 20:16:20.384836 29582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:16:20.401700 29582 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:16:20.403571 29582 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:16:20.410251 29582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2739ae47e0fd4fa49075618456af760d: Generated new TSK 0
I20260929 20:16:20.411193 29582 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
I20260929 20:16:20.458937 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:0
--local_ip_for_outbound_sockets=127.27.185.65
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:39527
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:39843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:20.789192 29597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:20.789508 29597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:20.789690 29597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:20.797436 29597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:20.797560 29597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:20.797708 29597 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:16:20.797842 29597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:16:20.808593 29597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:39843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.185.126:39527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:20.809937 29597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:20.811446 29597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:20.825667 29602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:20.825569 29603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:20.825599 29605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:20.826880 29597 server_base.cc:1034] running on GCE node
I20260929 20:16:20.827744 29597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:20.828747 29597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:20.830013 29597 hybrid_clock.cc:648] HybridClock initialized: now 1790712980829946 us; error 44 us; skew 500 ppm
I20260929 20:16:20.830351 29597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:20 dist-test-slave-4cx3 krb5kdc[29479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712980, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:20.833634 29597 init.cc:377] Logged in from keytab as kudu/127.27.185.65@KRBTEST.COM (short username kudu)
I20260929 20:16:20.835101 29597 webserver.cc:460] Webserver started at http://127.27.185.65:34819/ using document root <none> and password file <none>
I20260929 20:16:20.835865 29597 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:20.835999 29597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:20.836321 29597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:20.839242 29597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-0/data/instance:
uuid: "c72beba6466e431fb7bc3bfc3190ec74"
format_stamp: "Formatted at 2026-09-29 20:16:20 on dist-test-slave-4cx3"
server_key: "bd7bec7143e4fd045d9862a73fcd035e"
server_key_iv: "1c93a8f6b72b45cea74e25d5dbfb11fe"
server_key_version: "encryptionkey@0"
I20260929 20:16:20.840140 29597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance:
uuid: "c72beba6466e431fb7bc3bfc3190ec74"
format_stamp: "Formatted at 2026-09-29 20:16:20 on dist-test-slave-4cx3"
server_key: "bd7bec7143e4fd045d9862a73fcd035e"
server_key_iv: "1c93a8f6b72b45cea74e25d5dbfb11fe"
server_key_version: "encryptionkey@0"
I20260929 20:16:20.845432 29597 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:16:20.848976 29612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:20.850355 29597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:16:20.850565 29597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "c72beba6466e431fb7bc3bfc3190ec74"
format_stamp: "Formatted at 2026-09-29 20:16:20 on dist-test-slave-4cx3"
server_key: "bd7bec7143e4fd045d9862a73fcd035e"
server_key_iv: "1c93a8f6b72b45cea74e25d5dbfb11fe"
server_key_version: "encryptionkey@0"
I20260929 20:16:20.850787 29597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:20.874400 29597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:20.879197 29597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:20.879599 29597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:20.882218 29597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:16:20.882349 29597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:20.882508 29597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:16:20.882586 29597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:16:20 dist-test-slave-4cx3 krb5kdc[29479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712980, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:20.908355 29621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:20.883042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:55463 (local address 127.27.185.126:39527)
0929 20:16:20.883583 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:20.883589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:20.884889 (+ 1300us) server_negotiation.cc:407] Connection header received
0929 20:16:20.887165 (+ 2276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:20.887175 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:20.887278 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:20.887367 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:20.890064 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:20.891122 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:20.894135 (+ 3013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:20.894918 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:20.895088 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:20.900425 (+ 5337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:20.900434 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:20.900443 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:20.900492 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:20.904566 (+ 4074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:20.905531 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:20.905537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:20.905542 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:20.905701 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:20.906414 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:20.906421 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:20.906426 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:20.906781 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:20.906944 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:20.907757 (+ 813us) server_negotiation.cc:299] Negotiation successful
0929 20:16:20.907980 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":275,"threads_started":1}
I20260929 20:16:20.910043 29622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:20.883774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39527 (local address 127.27.185.65:55463)
0929 20:16:20.884683 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:20.884736 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:20.886784 (+ 2048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:20.888040 (+ 1256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:20.888053 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:20.888825 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:20.889829 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:20.889865 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:20.891281 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:20.891287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:20.893934 (+ 2647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:20.893951 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:20.895238 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:20.895252 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:20.895632 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:20.896636 (+ 1004us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:20.896671 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:20.900092 (+ 3421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:20.904754 (+ 4662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:20.904767 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:20.904790 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:20.905354 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:20.905877 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:20.905886 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:20.905894 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:20.906248 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:20.906935 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:20.906943 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:20.907314 (+ 371us) client_negotiation.cc:769] Sending connection context
0929 20:16:20.907666 (+ 352us) client_negotiation.cc:240] Negotiation successful
0929 20:16:20.908271 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":395,"threads_started":1}
I20260929 20:16:20.930852 29597 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:36961
I20260929 20:16:20.930882 29733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:36961 every 8 connection(s)
I20260929 20:16:20.932583 29597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:16:20.935139 29734 heartbeater.cc:347] Connected to a master server at 127.27.185.126:39527
I20260929 20:16:20.935372 29734 heartbeater.cc:464] Registering TS with master...
I20260929 20:16:20.936269 29734 heartbeater.cc:511] Master 127.27.185.126:39527 requested a full tablet report, sending...
I20260929 20:16:20.936553 29621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:20.916440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:38939 (local address 127.27.185.126:39527)
0929 20:16:20.916674 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:20.916679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:20.920587 (+ 3908us) server_negotiation.cc:407] Connection header received
0929 20:16:20.920944 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:20.920951 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:20.921049 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:20.921169 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:20.922615 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:20.923759 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:20.926622 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:20.927516 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:20.927677 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:20.930092 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:20.930099 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:20.930104 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:20.930142 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:20.933564 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:20.934467 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:20.934473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:20.934476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:20.934620 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:20.935069 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:20.935074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:20.935077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:20.935450 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:20.935606 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:20.936068 (+ 462us) server_negotiation.cc:299] Negotiation successful
0929 20:16:20.936237 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:16:20.936774 29703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:20.916038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39527 (local address 127.27.185.65:38939)
0929 20:16:20.920441 (+ 4403us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:20.920464 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:20.920731 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:20.921230 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:20.921240 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:20.922014 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:20.922438 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:20.922454 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:20.923957 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:20.923965 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:20.926398 (+ 2433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:20.926416 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:20.927752 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:20.927762 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:20.927990 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:20.928830 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:20.928863 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:20.929857 (+ 994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:20.933683 (+ 3826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:20.933688 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:20.933692 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:20.934337 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:20.934768 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:20.934771 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:20.934774 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:20.934955 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:20.935589 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:20.935594 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:20.935797 (+ 203us) client_negotiation.cc:769] Sending connection context
0929 20:16:20.935970 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 20:16:20.936311 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4167,"thread_start_us":3788,"threads_started":1}
I20260929 20:16:20.937551 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 29597
I20260929 20:16:20.937759 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance
I20260929 20:16:20.938608 29524 ts_manager.cc:194] Registered new tserver with Master: c72beba6466e431fb7bc3bfc3190ec74 (127.27.185.65:36961)
I20260929 20:16:20.938630 28389 external_mini_cluster.cc:1428] Setting key 9751c65b69ced72e77b2488d15e72974
I20260929 20:16:20.940358 29524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.65@KRBTEST.COM'} at 127.27.185.65:55463
WARNING: no policy specified for kudu/127.27.185.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
I20260929 20:16:20.980830 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:0
--local_ip_for_outbound_sockets=127.27.185.66
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:39527
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:39843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:21.366211 29738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:21.366521 29738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:21.366671 29738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:21.374466 29738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:21.374583 29738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:21.374732 29738 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:16:21.374843 29738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:16:21.385191 29738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:39843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.185.126:39527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:21.386523 29738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:21.387898 29738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:21.401183 29743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:21.401233 29746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:21.401263 29744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:21.402117 29738 server_base.cc:1034] running on GCE node
I20260929 20:16:21.402690 29738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:21.403640 29738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:21.404861 29738 hybrid_clock.cc:648] HybridClock initialized: now 1790712981404778 us; error 55 us; skew 500 ppm
I20260929 20:16:21.405192 29738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:21 dist-test-slave-4cx3 krb5kdc[29479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712981, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:21.408566 29738 init.cc:377] Logged in from keytab as kudu/127.27.185.66@KRBTEST.COM (short username kudu)
I20260929 20:16:21.410270 29738 webserver.cc:460] Webserver started at http://127.27.185.66:44045/ using document root <none> and password file <none>
I20260929 20:16:21.411173 29738 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:21.411325 29738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:21.411703 29738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:21.414999 29738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-1/data/instance:
uuid: "8e53af7f316f426da3c6262fbd4c43ca"
format_stamp: "Formatted at 2026-09-29 20:16:21 on dist-test-slave-4cx3"
server_key: "54bcf4f29cf50940fe22d227d3b6b0ca"
server_key_iv: "fda2068223ee2d7cd5802b714901808f"
server_key_version: "encryptionkey@0"
I20260929 20:16:21.415900 29738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance:
uuid: "8e53af7f316f426da3c6262fbd4c43ca"
format_stamp: "Formatted at 2026-09-29 20:16:21 on dist-test-slave-4cx3"
server_key: "54bcf4f29cf50940fe22d227d3b6b0ca"
server_key_iv: "fda2068223ee2d7cd5802b714901808f"
server_key_version: "encryptionkey@0"
I20260929 20:16:21.421675 29738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:16:21.425361 29753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:21.426786 29738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:16:21.426971 29738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "8e53af7f316f426da3c6262fbd4c43ca"
format_stamp: "Formatted at 2026-09-29 20:16:21 on dist-test-slave-4cx3"
server_key: "54bcf4f29cf50940fe22d227d3b6b0ca"
server_key_iv: "fda2068223ee2d7cd5802b714901808f"
server_key_version: "encryptionkey@0"
I20260929 20:16:21.427222 29738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:21.442476 29738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:21.450554 29738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:21.451020 29738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:21.455188 29738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:16:21.455360 29738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:21.455768 29738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:16:21.455894 29738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:16:21 dist-test-slave-4cx3 krb5kdc[29479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712981, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:21.499495 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:21.456723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:39753 (local address 127.27.185.126:39527)
0929 20:16:21.457682 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:21.457690 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:21.459882 (+ 2192us) server_negotiation.cc:407] Connection header received
0929 20:16:21.463002 (+ 3120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:21.463010 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:21.463165 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:21.463351 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:21.467444 (+ 4093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:21.469228 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:21.473725 (+ 4497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:21.474799 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:21.475032 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:21.484320 (+ 9288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:21.484335 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:21.484339 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:21.484394 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:21.491303 (+ 6909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:21.493661 (+ 2358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:21.493672 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:21.493689 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:21.494017 (+ 328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:21.495169 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:21.495178 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:21.495194 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:21.495964 (+ 770us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:21.496269 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:21.498038 (+ 1769us) server_negotiation.cc:299] Negotiation successful
0929 20:16:21.498536 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":644,"threads_started":1}
I20260929 20:16:21.502646 29763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:21.458051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39527 (local address 127.27.185.66:39753)
0929 20:16:21.459566 (+ 1515us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:21.459684 (+ 118us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:21.462526 (+ 2842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:21.463848 (+ 1322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:21.463865 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:21.465179 (+ 1314us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:21.467025 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:21.467150 (+ 125us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:21.469539 (+ 2389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:21.469554 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:21.473473 (+ 3919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:21.473495 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:21.475310 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:21.475328 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:21.475940 (+ 612us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:21.477705 (+ 1765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:21.477790 (+ 85us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:21.483748 (+ 5958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:21.491724 (+ 7976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:21.491743 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:21.491785 (+ 42us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:21.493277 (+ 1492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:21.494265 (+ 988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:21.494296 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:21.494303 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:21.494916 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:21.496223 (+ 1307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:21.496241 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:21.497218 (+ 977us) client_negotiation.cc:769] Sending connection context
0929 20:16:21.497841 (+ 623us) client_negotiation.cc:240] Negotiation successful
0929 20:16:21.499070 (+ 1229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":918,"thread_start_us":616,"threads_started":1}
I20260929 20:16:21.527093 29874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:43141 every 8 connection(s)
I20260929 20:16:21.527093 29738 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:43141
I20260929 20:16:21.529484 29738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:16:21.533094 29875 heartbeater.cc:347] Connected to a master server at 127.27.185.126:39527
I20260929 20:16:21.533285 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 29738
I20260929 20:16:21.533535 29875 heartbeater.cc:464] Registering TS with master...
I20260929 20:16:21.533601 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance
I20260929 20:16:21.534644 28389 external_mini_cluster.cc:1428] Setting key 7e96ded8b6df236ad408f80df99c9ae0
I20260929 20:16:21.534974 29875 heartbeater.cc:511] Master 127.27.185.126:39527 requested a full tablet report, sending...
I20260929 20:16:21.537957 29525 ts_manager.cc:194] Registered new tserver with Master: 8e53af7f316f426da3c6262fbd4c43ca (127.27.185.66:43141)
I20260929 20:16:21.540088 29525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.66@KRBTEST.COM'} at 127.27.185.66:39753
I20260929 20:16:21.545161 29864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:21.515731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39527 (local address 127.27.185.66:33689)
0929 20:16:21.517865 (+ 2134us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:21.517919 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:21.518289 (+ 370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:21.519544 (+ 1255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:21.519557 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:21.520626 (+ 1069us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:21.521382 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:21.521410 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:21.523770 (+ 2360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:21.523776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:21.526153 (+ 2377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:21.526197 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:21.527909 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:21.527915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:21.528129 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:21.529026 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:21.529058 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:21.530691 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:21.540185 (+ 9494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:21.540217 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:21.540225 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:21.541260 (+ 1035us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:21.542131 (+ 871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:21.542151 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:21.542158 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:21.542460 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:21.543836 (+ 1376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:21.543850 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:21.544107 (+ 257us) client_negotiation.cc:769] Sending connection context
0929 20:16:21.544326 (+ 219us) client_negotiation.cc:240] Negotiation successful
0929 20:16:21.544690 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1897,"thread_start_us":1487,"threads_started":1}
I20260929 20:16:21.545751 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:21.515057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:33689 (local address 127.27.185.126:39527)
0929 20:16:21.515365 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:21.515426 (+ 61us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:21.518219 (+ 2793us) server_negotiation.cc:407] Connection header received
0929 20:16:21.518755 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:21.518765 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:21.519148 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:21.521676 (+ 2528us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:21.521942 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:21.523562 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:21.526436 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:21.527604 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:21.527866 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:21.531118 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:21.531127 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:21.531132 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:21.531208 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:21.539954 (+ 8746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:21.541527 (+ 1573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:21.541552 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:21.541560 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:21.541821 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:21.542688 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:21.542702 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:21.542710 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:21.543586 (+ 876us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:21.543862 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:21.544516 (+ 654us) server_negotiation.cc:299] Negotiation successful
0929 20:16:21.544961 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
WARNING: no policy specified for kudu/127.27.185.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
I20260929 20:16:21.595759 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:0
--local_ip_for_outbound_sockets=127.27.185.67
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:39527
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:39843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:16:21.944242 29734 heartbeater.cc:503] Master 127.27.185.126:39527 was elected leader, sending a full tablet report...
W20260929 20:16:22.125416 29879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:22.125917 29879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:22.126068 29879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:22.138304 29879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:22.138494 29879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:22.138628 29879 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:16:22.138752 29879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:16:22.154778 29879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:39843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.185.126:39527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:22.157090 29879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:22.158547 29879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:22.176043 29884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:22.176112 29885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:22.176772 29887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:22.178277 29879 server_base.cc:1034] running on GCE node
I20260929 20:16:22.179311 29879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:22.181026 29879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:22.182438 29879 hybrid_clock.cc:648] HybridClock initialized: now 1790712982182322 us; error 102 us; skew 500 ppm
I20260929 20:16:22.182917 29879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:22 dist-test-slave-4cx3 krb5kdc[29479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712982, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:22.188143 29879 init.cc:377] Logged in from keytab as kudu/127.27.185.67@KRBTEST.COM (short username kudu)
I20260929 20:16:22.190653 29879 webserver.cc:460] Webserver started at http://127.27.185.67:43311/ using document root <none> and password file <none>
I20260929 20:16:22.191869 29879 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:22.192095 29879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:22.192622 29879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:22.197824 29879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-2/data/instance:
uuid: "e10f9c7cfdb84084bc4d25713c4d8325"
format_stamp: "Formatted at 2026-09-29 20:16:22 on dist-test-slave-4cx3"
server_key: "bbc83058eec041556ab8cc68375beb41"
server_key_iv: "b447fdcdd78bc99105ad08733aa14b27"
server_key_version: "encryptionkey@0"
I20260929 20:16:22.199249 29879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance:
uuid: "e10f9c7cfdb84084bc4d25713c4d8325"
format_stamp: "Formatted at 2026-09-29 20:16:22 on dist-test-slave-4cx3"
server_key: "bbc83058eec041556ab8cc68375beb41"
server_key_iv: "b447fdcdd78bc99105ad08733aa14b27"
server_key_version: "encryptionkey@0"
I20260929 20:16:22.208633 29879 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.001s
I20260929 20:16:22.214781 29894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:22.217069 29879 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260929 20:16:22.217339 29879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "e10f9c7cfdb84084bc4d25713c4d8325"
format_stamp: "Formatted at 2026-09-29 20:16:22 on dist-test-slave-4cx3"
server_key: "bbc83058eec041556ab8cc68375beb41"
server_key_iv: "b447fdcdd78bc99105ad08733aa14b27"
server_key_version: "encryptionkey@0"
I20260929 20:16:22.217700 29879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:22.246325 29879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:22.253005 29879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:22.253485 29879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:22.258085 29879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:16:22.258240 29879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:22.258473 29879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:16:22.258605 29879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:16:22 dist-test-slave-4cx3 krb5kdc[29479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712982, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:22.298878 29903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:22.258818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:43897 (local address 127.27.185.126:39527)
0929 20:16:22.260172 (+ 1354us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:22.260192 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:22.261941 (+ 1749us) server_negotiation.cc:407] Connection header received
0929 20:16:22.267190 (+ 5249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:22.267211 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:22.267438 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:22.267652 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:22.271351 (+ 3699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.273606 (+ 2255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.277691 (+ 4085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.278702 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.278951 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:22.285789 (+ 6838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:22.285799 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:22.285818 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:22.285915 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:22.291499 (+ 5584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:22.293216 (+ 1717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:22.293232 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:22.293239 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:22.293523 (+ 284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:22.294517 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:22.294530 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:22.294540 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:22.295482 (+ 942us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:22.295792 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:22.298060 (+ 2268us) server_negotiation.cc:299] Negotiation successful
0929 20:16:22.298346 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1156,"thread_start_us":971,"threads_started":1}
I20260929 20:16:22.301715 29904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:22.260238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39527 (local address 127.27.185.67:43897)
0929 20:16:22.261619 (+ 1381us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:22.261721 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:22.266464 (+ 4743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:22.268234 (+ 1770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:22.268248 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:22.269281 (+ 1033us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:22.270972 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:22.271010 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.273905 (+ 2895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.273926 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:22.277429 (+ 3503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:22.277467 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.279116 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.279124 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:22.279583 (+ 459us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:22.280928 (+ 1345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:22.280978 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:22.285407 (+ 4429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:22.291763 (+ 6356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:22.291811 (+ 48us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:22.291851 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:22.292915 (+ 1064us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:22.293730 (+ 815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:22.293734 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:22.293737 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:22.294259 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:22.295991 (+ 1732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:22.296013 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:22.297197 (+ 1184us) client_negotiation.cc:769] Sending connection context
0929 20:16:22.297848 (+ 651us) client_negotiation.cc:240] Negotiation successful
0929 20:16:22.299789 (+ 1941us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"thread_start_us":500,"threads_started":1}
I20260929 20:16:22.340859 29903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:22.312988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:54549 (local address 127.27.185.126:39527)
0929 20:16:22.313310 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:22.313320 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:22.314771 (+ 1451us) server_negotiation.cc:407] Connection header received
0929 20:16:22.315121 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:22.315128 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:22.315237 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:22.315444 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:22.318051 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.319725 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.323157 (+ 3432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.324281 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.324508 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:22.328265 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:22.328272 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:22.328288 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:22.328357 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:22.334857 (+ 6500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:22.336633 (+ 1776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:22.336640 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:22.336644 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:22.336897 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:22.337769 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:22.337776 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:22.337780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:22.338405 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:22.338612 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:22.339793 (+ 1181us) server_negotiation.cc:299] Negotiation successful
0929 20:16:22.340267 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260929 20:16:22.342972 29978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:22.313057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39527 (local address 127.27.185.67:54549)
0929 20:16:22.314627 (+ 1570us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:22.314652 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:22.314900 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:22.315456 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:22.315460 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:22.316768 (+ 1308us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:22.317524 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:22.317563 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.319920 (+ 2357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.319939 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:22.322851 (+ 2912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:22.322884 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.324571 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.324579 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:22.324867 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:22.325995 (+ 1128us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:22.326046 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:22.327926 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:22.335118 (+ 7192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:22.335133 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:22.335138 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:22.336403 (+ 1265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:22.337187 (+ 784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:22.337204 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:22.337208 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:22.337537 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:22.338763 (+ 1226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:22.338770 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:22.339188 (+ 418us) client_negotiation.cc:769] Sending connection context
0929 20:16:22.341206 (+ 2018us) client_negotiation.cc:240] Negotiation successful
0929 20:16:22.341928 (+ 722us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"thread_start_us":564,"threads_started":1}
I20260929 20:16:22.352349 29879 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:40393
I20260929 20:16:22.352417 30015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:40393 every 8 connection(s)
I20260929 20:16:22.354701 29879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:16:22.362581 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 29879
I20260929 20:16:22.362954 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance
I20260929 20:16:22.358582 30016 heartbeater.cc:347] Connected to a master server at 127.27.185.126:39527
I20260929 20:16:22.363463 30016 heartbeater.cc:464] Registering TS with master...
I20260929 20:16:22.364188 28389 external_mini_cluster.cc:1428] Setting key 91e21a72c4ea6b7f4092e6421d71c16b
I20260929 20:16:22.365295 30016 heartbeater.cc:511] Master 127.27.185.126:39527 requested a full tablet report, sending...
I20260929 20:16:22.368566 29525 ts_manager.cc:194] Registered new tserver with Master: e10f9c7cfdb84084bc4d25713c4d8325 (127.27.185.67:40393)
I20260929 20:16:22.370682 29525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.67@KRBTEST.COM'} at 127.27.185.67:43897
I20260929 20:16:22.371673 28389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 20:16:22 dist-test-slave-4cx3 krb5kdc[29479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712982, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 20:16:22 dist-test-slave-4cx3 krb5kdc[29479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712982, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:22.445541 29903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:22.421727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41518 (local address 127.27.185.126:39527)
0929 20:16:22.421968 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:22.421981 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:22.422530 (+ 549us) server_negotiation.cc:407] Connection header received
0929 20:16:22.422844 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:22.422851 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:22.422978 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:22.423189 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:22.426095 (+ 2906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.427197 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.429407 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.430055 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.430334 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:22.437001 (+ 6667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:22.437010 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:22.437013 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:22.437083 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:22.441120 (+ 4037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:22.442196 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:22.442212 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:22.442220 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:22.442422 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:22.443010 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:22.443026 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:22.443030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:22.443558 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:22.444012 (+ 454us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:22.444544 (+ 532us) server_negotiation.cc:299] Negotiation successful
0929 20:16:22.444927 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 20:16:22.452152 29525 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:16:22.464429 29525 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:16:22.505728 30031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:22.486000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48298 (local address 127.27.185.66:43141)
0929 20:16:22.489003 (+ 3003us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:22.489031 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:22.489114 (+ 83us) server_negotiation.cc:407] Connection header received
0929 20:16:22.489374 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:22.489398 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:22.490006 (+ 608us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:22.492177 (+ 2171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:16:22.492650 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.495614 (+ 2964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.501410 (+ 5796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.503820 (+ 2410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.504090 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:22.504483 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:22.504951 (+ 468us) server_negotiation.cc:299] Negotiation successful
0929 20:16:22.505214 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2575,"thread_start_us":2283,"threads_started":1}
I20260929 20:16:22.506130 30030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:22.485690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:43141 (local address 127.0.0.1:48298)
0929 20:16:22.487332 (+ 1642us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:22.487435 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:22.487777 (+ 342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:22.490532 (+ 2755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:22.490556 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:22.490627 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:16:22.491695 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:22.491734 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.495994 (+ 4260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.496007 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:22.501079 (+ 5072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:22.501110 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.504153 (+ 3043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.504174 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:22.504596 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:22.504646 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 20:16:22.504840 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 20:16:22.505231 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1186,"mutex_wait_us":11,"thread_start_us":955,"threads_started":1}
I20260929 20:16:22.514077 29804 tablet_service.cc:1467] Processing CreateTablet for tablet c69a720e5f7e4ef68ae42df7535a98cf (DEFAULT_TABLE table=test-table [id=5cb6a5ea8a404cc09c8fdc7d610b7678]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:16:22.517196 29804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c69a720e5f7e4ef68ae42df7535a98cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:22.519317 30033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:22.489690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51080 (local address 127.27.185.67:40393)
0929 20:16:22.501167 (+ 11477us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:22.501182 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:22.502147 (+ 965us) server_negotiation.cc:407] Connection header received
0929 20:16:22.502308 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:22.502316 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:22.502702 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:22.503024 (+ 322us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:16:22.504674 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.508066 (+ 3392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.511783 (+ 3717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.514989 (+ 3206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.515226 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:22.515701 (+ 475us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:22.516638 (+ 937us) server_negotiation.cc:299] Negotiation successful
0929 20:16:22.516996 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11194,"thread_start_us":10915,"threads_started":1}
I20260929 20:16:22.519528 30034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:22.491407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:40393 (local address 127.0.0.1:51080)
0929 20:16:22.501693 (+ 10286us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:22.501731 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:22.501925 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:22.503220 (+ 1295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:22.503228 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:22.503280 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:16:22.504260 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:22.504285 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.508287 (+ 4002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.508429 (+ 142us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:22.511416 (+ 2987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:22.511436 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.515450 (+ 4014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.515461 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:22.515818 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:22.515901 (+ 83us) client_negotiation.cc:769] Sending connection context
0929 20:16:22.516052 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 20:16:22.518344 (+ 2292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10079,"mutex_wait_us":97,"thread_start_us":9796,"threads_started":1}
I20260929 20:16:22.521485 30029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:22.485870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:36961 (local address 127.0.0.1:50024)
0929 20:16:22.500432 (+ 14562us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:22.500480 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:22.501259 (+ 779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:22.503037 (+ 1778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:22.503041 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:22.503079 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:16:22.504134 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:22.504173 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.507051 (+ 2878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.507070 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:22.517618 (+ 10548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:22.517646 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.520866 (+ 3220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.520875 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:22.521054 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:22.521078 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 20:16:22.521136 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 20:16:22.521232 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14179,"mutex_wait_us":6,"thread_start_us":14879,"threads_started":1}
I20260929 20:16:22.522334 30032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:22.486768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50024 (local address 127.27.185.65:36961)
0929 20:16:22.499595 (+ 12827us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:22.499613 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:22.500697 (+ 1084us) server_negotiation.cc:407] Connection header received
0929 20:16:22.501641 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:22.501662 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:22.502172 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:22.502514 (+ 342us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:16:22.504495 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.506732 (+ 2237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.517970 (+ 11238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.520519 (+ 2549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.520634 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:22.521782 (+ 1148us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:22.521868 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 20:16:22.522049 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12506,"thread_start_us":12814,"threads_started":1}
I20260929 20:16:22.524621 29945 tablet_service.cc:1467] Processing CreateTablet for tablet c69a720e5f7e4ef68ae42df7535a98cf (DEFAULT_TABLE table=test-table [id=5cb6a5ea8a404cc09c8fdc7d610b7678]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:16:22.525925 29663 tablet_service.cc:1467] Processing CreateTablet for tablet c69a720e5f7e4ef68ae42df7535a98cf (DEFAULT_TABLE table=test-table [id=5cb6a5ea8a404cc09c8fdc7d610b7678]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:16:22.526851 29945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c69a720e5f7e4ef68ae42df7535a98cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:22.527383 29663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c69a720e5f7e4ef68ae42df7535a98cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:22.539084 30035 tablet_bootstrap.cc:492] T c69a720e5f7e4ef68ae42df7535a98cf P 8e53af7f316f426da3c6262fbd4c43ca: Bootstrap starting.
I20260929 20:16:22.541307 30036 tablet_bootstrap.cc:492] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74: Bootstrap starting.
I20260929 20:16:22.542769 30037 tablet_bootstrap.cc:492] T c69a720e5f7e4ef68ae42df7535a98cf P e10f9c7cfdb84084bc4d25713c4d8325: Bootstrap starting.
I20260929 20:16:22.542927 30035 tablet_bootstrap.cc:654] T c69a720e5f7e4ef68ae42df7535a98cf P 8e53af7f316f426da3c6262fbd4c43ca: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:22.544080 29875 heartbeater.cc:503] Master 127.27.185.126:39527 was elected leader, sending a full tablet report...
I20260929 20:16:22.544354 30035 log.cc:824] T c69a720e5f7e4ef68ae42df7535a98cf P 8e53af7f316f426da3c6262fbd4c43ca: Log is configured to *not* fsync() on all Append() calls
I20260929 20:16:22.546479 30037 tablet_bootstrap.cc:654] T c69a720e5f7e4ef68ae42df7535a98cf P e10f9c7cfdb84084bc4d25713c4d8325: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:22.546734 30036 tablet_bootstrap.cc:654] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:22.547904 30036 log.cc:824] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74: Log is configured to *not* fsync() on all Append() calls
I20260929 20:16:22.547895 30037 log.cc:824] T c69a720e5f7e4ef68ae42df7535a98cf P e10f9c7cfdb84084bc4d25713c4d8325: Log is configured to *not* fsync() on all Append() calls
I20260929 20:16:22.549651 30035 tablet_bootstrap.cc:492] T c69a720e5f7e4ef68ae42df7535a98cf P 8e53af7f316f426da3c6262fbd4c43ca: No bootstrap required, opened a new log
I20260929 20:16:22.549933 30035 ts_tablet_manager.cc:1397] T c69a720e5f7e4ef68ae42df7535a98cf P 8e53af7f316f426da3c6262fbd4c43ca: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260929 20:16:22.550680 30036 tablet_bootstrap.cc:492] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74: No bootstrap required, opened a new log
I20260929 20:16:22.550945 30036 ts_tablet_manager.cc:1397] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260929 20:16:22.551043 30037 tablet_bootstrap.cc:492] T c69a720e5f7e4ef68ae42df7535a98cf P e10f9c7cfdb84084bc4d25713c4d8325: No bootstrap required, opened a new log
I20260929 20:16:22.551290 30037 ts_tablet_manager.cc:1397] T c69a720e5f7e4ef68ae42df7535a98cf P e10f9c7cfdb84084bc4d25713c4d8325: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260929 20:16:22.558706 30036 raft_consensus.cc:348] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } }
I20260929 20:16:22.559090 30036 raft_consensus.cc:374] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:22.559199 30036 raft_consensus.cc:729] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c72beba6466e431fb7bc3bfc3190ec74, State: Initialized, Role: FOLLOWER
I20260929 20:16:22.559883 30036 consensus_queue.cc:260] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } }
I20260929 20:16:22.560626 30035 raft_consensus.cc:348] T c69a720e5f7e4ef68ae42df7535a98cf P 8e53af7f316f426da3c6262fbd4c43ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } }
I20260929 20:16:22.561070 30035 raft_consensus.cc:374] T c69a720e5f7e4ef68ae42df7535a98cf P 8e53af7f316f426da3c6262fbd4c43ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:22.561160 30035 raft_consensus.cc:729] T c69a720e5f7e4ef68ae42df7535a98cf P 8e53af7f316f426da3c6262fbd4c43ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e53af7f316f426da3c6262fbd4c43ca, State: Initialized, Role: FOLLOWER
I20260929 20:16:22.561741 30035 consensus_queue.cc:260] T c69a720e5f7e4ef68ae42df7535a98cf P 8e53af7f316f426da3c6262fbd4c43ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } }
I20260929 20:16:22.561936 30037 raft_consensus.cc:348] T c69a720e5f7e4ef68ae42df7535a98cf P e10f9c7cfdb84084bc4d25713c4d8325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } }
I20260929 20:16:22.562388 30037 raft_consensus.cc:374] T c69a720e5f7e4ef68ae42df7535a98cf P e10f9c7cfdb84084bc4d25713c4d8325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:22.562505 30037 raft_consensus.cc:729] T c69a720e5f7e4ef68ae42df7535a98cf P e10f9c7cfdb84084bc4d25713c4d8325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e10f9c7cfdb84084bc4d25713c4d8325, State: Initialized, Role: FOLLOWER
I20260929 20:16:22.561836 30036 ts_tablet_manager.cc:1428] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260929 20:16:22.563203 30037 consensus_queue.cc:260] T c69a720e5f7e4ef68ae42df7535a98cf P e10f9c7cfdb84084bc4d25713c4d8325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } }
I20260929 20:16:22.565140 30035 ts_tablet_manager.cc:1428] T c69a720e5f7e4ef68ae42df7535a98cf P 8e53af7f316f426da3c6262fbd4c43ca: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260929 20:16:22.566978 30016 heartbeater.cc:503] Master 127.27.185.126:39527 was elected leader, sending a full tablet report...
I20260929 20:16:22.567780 30037 ts_tablet_manager.cc:1428] T c69a720e5f7e4ef68ae42df7535a98cf P e10f9c7cfdb84084bc4d25713c4d8325: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
W20260929 20:16:22.607692 30017 tablet.cc:2369] T c69a720e5f7e4ef68ae42df7535a98cf P e10f9c7cfdb84084bc4d25713c4d8325: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:16:22.653352 30041 raft_consensus.cc:482] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:16:22.653707 30041 raft_consensus.cc:504] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } }
I20260929 20:16:22.655562 30041 leader_election.cc:288] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e53af7f316f426da3c6262fbd4c43ca (127.27.185.66:43141), e10f9c7cfdb84084bc4d25713c4d8325 (127.27.185.67:40393)
I20260929 20:16:22.664177 30031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:22.655983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:58745 (local address 127.27.185.66:43141)
0929 20:16:22.656235 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:22.656241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:22.656520 (+ 279us) server_negotiation.cc:407] Connection header received
0929 20:16:22.656788 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:22.656795 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:22.656879 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:22.656952 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:16:22.657889 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.658907 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.661672 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.662926 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.663107 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:22.663183 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:22.663747 (+ 564us) server_negotiation.cc:299] Negotiation successful
0929 20:16:22.663886 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 20:16:22.664173 30033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:22.656799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:58141 (local address 127.27.185.67:40393)
0929 20:16:22.656995 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:22.657001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:22.657216 (+ 215us) server_negotiation.cc:407] Connection header received
0929 20:16:22.657333 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:22.657340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:22.657423 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:22.657532 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:16:22.658153 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.659078 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.661665 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.662926 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.663108 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:22.663183 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:22.663747 (+ 564us) server_negotiation.cc:299] Negotiation successful
0929 20:16:22.663886 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:16:22.664414 30045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:22.656466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:40393 (local address 127.27.185.65:58141)
0929 20:16:22.657106 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:22.657124 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:22.657229 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:22.657542 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:22.657545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:22.657562 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:16:22.657960 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:22.657995 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.659228 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.659234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:22.661488 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:22.661504 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.663217 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.663223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:22.663489 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:22.663596 (+ 107us) client_negotiation.cc:769] Sending connection context
0929 20:16:22.663697 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 20:16:22.663886 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":141,"thread_start_us":320,"threads_started":1}
I20260929 20:16:22.664777 30044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:22.655765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:43141 (local address 127.27.185.65:58745)
0929 20:16:22.656364 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:22.656381 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:22.656579 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:22.657191 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:22.657197 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:22.657289 (+ 92us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:16:22.657723 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:22.657735 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.659045 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.659053 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:22.661487 (+ 2434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:22.661504 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.663193 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.663204 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:22.663488 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:22.663596 (+ 108us) client_negotiation.cc:769] Sending connection context
0929 20:16:22.663697 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 20:16:22.663886 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":147,"spinlock_wait_cycles":27008,"thread_start_us":321,"threads_started":1}
I20260929 20:16:22.665517 29965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c69a720e5f7e4ef68ae42df7535a98cf" candidate_uuid: "c72beba6466e431fb7bc3bfc3190ec74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" is_pre_election: true
I20260929 20:16:22.665688 29824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c69a720e5f7e4ef68ae42df7535a98cf" candidate_uuid: "c72beba6466e431fb7bc3bfc3190ec74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e53af7f316f426da3c6262fbd4c43ca" is_pre_election: true
I20260929 20:16:22.666112 29965 raft_consensus.cc:2454] T c69a720e5f7e4ef68ae42df7535a98cf P e10f9c7cfdb84084bc4d25713c4d8325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c72beba6466e431fb7bc3bfc3190ec74 in term 0.
I20260929 20:16:22.666224 29824 raft_consensus.cc:2454] T c69a720e5f7e4ef68ae42df7535a98cf P 8e53af7f316f426da3c6262fbd4c43ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c72beba6466e431fb7bc3bfc3190ec74 in term 0.
I20260929 20:16:22.666826 29613 leader_election.cc:302] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c72beba6466e431fb7bc3bfc3190ec74, e10f9c7cfdb84084bc4d25713c4d8325; no voters:
I20260929 20:16:22.667260 30041 raft_consensus.cc:2781] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:16:22.667485 30041 raft_consensus.cc:482] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:16:22.667608 30041 raft_consensus.cc:3037] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:22.669857 30041 raft_consensus.cc:504] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } }
I20260929 20:16:22.670603 30041 leader_election.cc:288] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74 [CANDIDATE]: Term 1 election: Requested vote from peers 8e53af7f316f426da3c6262fbd4c43ca (127.27.185.66:43141), e10f9c7cfdb84084bc4d25713c4d8325 (127.27.185.67:40393)
I20260929 20:16:22.671136 29824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c69a720e5f7e4ef68ae42df7535a98cf" candidate_uuid: "c72beba6466e431fb7bc3bfc3190ec74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e53af7f316f426da3c6262fbd4c43ca"
I20260929 20:16:22.671217 29965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c69a720e5f7e4ef68ae42df7535a98cf" candidate_uuid: "c72beba6466e431fb7bc3bfc3190ec74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e10f9c7cfdb84084bc4d25713c4d8325"
I20260929 20:16:22.671353 29824 raft_consensus.cc:3037] T c69a720e5f7e4ef68ae42df7535a98cf P 8e53af7f316f426da3c6262fbd4c43ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:22.671500 29965 raft_consensus.cc:3037] T c69a720e5f7e4ef68ae42df7535a98cf P e10f9c7cfdb84084bc4d25713c4d8325 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:22.673733 29824 raft_consensus.cc:2454] T c69a720e5f7e4ef68ae42df7535a98cf P 8e53af7f316f426da3c6262fbd4c43ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c72beba6466e431fb7bc3bfc3190ec74 in term 1.
I20260929 20:16:22.673743 29965 raft_consensus.cc:2454] T c69a720e5f7e4ef68ae42df7535a98cf P e10f9c7cfdb84084bc4d25713c4d8325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c72beba6466e431fb7bc3bfc3190ec74 in term 1.
I20260929 20:16:22.674144 29613 leader_election.cc:302] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c72beba6466e431fb7bc3bfc3190ec74, e10f9c7cfdb84084bc4d25713c4d8325; no voters:
I20260929 20:16:22.674453 30041 raft_consensus.cc:2781] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:16:22.674908 30041 raft_consensus.cc:686] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74 [term 1 LEADER]: Becoming Leader. State: Replica: c72beba6466e431fb7bc3bfc3190ec74, State: Running, Role: LEADER
I20260929 20:16:22.675490 30041 consensus_queue.cc:237] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } }
I20260929 20:16:22.680370 29524 catalog_manager.cc:5495] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74 reported cstate change: term changed from 0 to 1, leader changed from <none> to c72beba6466e431fb7bc3bfc3190ec74 (127.27.185.65). New cstate: current_term: 1 leader_uuid: "c72beba6466e431fb7bc3bfc3190ec74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } health_report { overall_health: UNKNOWN } } }
W20260929 20:16:22.685856 29735 tablet.cc:2369] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:16:22.714879 30032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:22.709034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50034 (local address 127.27.185.65:36961)
0929 20:16:22.709248 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:22.709254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:22.709413 (+ 159us) server_negotiation.cc:407] Connection header received
0929 20:16:22.709777 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:22.709782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:22.709872 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:22.710029 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:16:22.710378 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.711125 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.712649 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.713061 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.713188 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:22.713720 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:16:22.714235 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:16:22.714390 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:22.714525 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 20:16:22.714610 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:16:22.723637 29965 raft_consensus.cc:1261] T c69a720e5f7e4ef68ae42df7535a98cf P e10f9c7cfdb84084bc4d25713c4d8325 [term 1 FOLLOWER]: Refusing update from remote peer c72beba6466e431fb7bc3bfc3190ec74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:16:22.723636 29824 raft_consensus.cc:1261] T c69a720e5f7e4ef68ae42df7535a98cf P 8e53af7f316f426da3c6262fbd4c43ca [term 1 FOLLOWER]: Refusing update from remote peer c72beba6466e431fb7bc3bfc3190ec74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:16:22.724789 30046 consensus_queue.cc:1035] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:16:22.725178 30041 consensus_queue.cc:1035] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:16:22.742062 30050 mvcc.cc:205] Tried to move back new op lower bound from 7334760377224613888 to 7334760377042862080. Current Snapshot: MvccSnapshot[applied={T|T < 7334760377224613888 or (T in {7334760377224613888})}]
I20260929 20:16:22.757354 29525 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:16:22.757689 29525 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:16:22.762373 29525 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2739ae47e0fd4fa49075618456af760d: Sending DeleteTablet for 3 replicas of tablet c69a720e5f7e4ef68ae42df7535a98cf
I20260929 20:16:22.763739 29663 tablet_service.cc:1514] Processing DeleteTablet for tablet c69a720e5f7e4ef68ae42df7535a98cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:16:22 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:50024
I20260929 20:16:22.763805 29945 tablet_service.cc:1514] Processing DeleteTablet for tablet c69a720e5f7e4ef68ae42df7535a98cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:16:22 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:51080
I20260929 20:16:22.764564 29804 tablet_service.cc:1514] Processing DeleteTablet for tablet c69a720e5f7e4ef68ae42df7535a98cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:16:22 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:48298
I20260929 20:16:22.764810 30059 tablet_replica.cc:330] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74: stopping tablet replica
I20260929 20:16:22.765406 30059 raft_consensus.cc:2229] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:16:22.766412 30059 raft_consensus.cc:2258] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:16:22.766350 29524 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:16:22.767108 29524 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:16:22.776937 30061 tablet_replica.cc:330] T c69a720e5f7e4ef68ae42df7535a98cf P 8e53af7f316f426da3c6262fbd4c43ca: stopping tablet replica
I20260929 20:16:22.777498 30061 raft_consensus.cc:2229] T c69a720e5f7e4ef68ae42df7535a98cf P 8e53af7f316f426da3c6262fbd4c43ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:16:22.778055 30061 raft_consensus.cc:2258] T c69a720e5f7e4ef68ae42df7535a98cf P 8e53af7f316f426da3c6262fbd4c43ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:16:22.778187 29663 tablet_service.cc:1467] Processing CreateTablet for tablet d4f7b342ca5c4d25a626d8b716e5fc17 (DEFAULT_TABLE table=test-table [id=af78a6cb44eb42bcb254828bb14a3e84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:16:22.778864 29663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4f7b342ca5c4d25a626d8b716e5fc17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:22.779161 30059 ts_tablet_manager.cc:1905] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:16:22.779582 30060 tablet_replica.cc:330] T c69a720e5f7e4ef68ae42df7535a98cf P e10f9c7cfdb84084bc4d25713c4d8325: stopping tablet replica
I20260929 20:16:22.780041 30060 raft_consensus.cc:2229] T c69a720e5f7e4ef68ae42df7535a98cf P e10f9c7cfdb84084bc4d25713c4d8325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:16:22.780283 30061 ts_tablet_manager.cc:1905] T c69a720e5f7e4ef68ae42df7535a98cf P 8e53af7f316f426da3c6262fbd4c43ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:16:22.780562 30060 raft_consensus.cc:2258] T c69a720e5f7e4ef68ae42df7535a98cf P e10f9c7cfdb84084bc4d25713c4d8325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:16:22.783850 30060 ts_tablet_manager.cc:1905] T c69a720e5f7e4ef68ae42df7535a98cf P e10f9c7cfdb84084bc4d25713c4d8325: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:16:22.784536 29804 tablet_service.cc:1467] Processing CreateTablet for tablet d4f7b342ca5c4d25a626d8b716e5fc17 (DEFAULT_TABLE table=test-table [id=af78a6cb44eb42bcb254828bb14a3e84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:16:22.785040 29804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4f7b342ca5c4d25a626d8b716e5fc17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:22.786921 30059 ts_tablet_manager.cc:1918] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:16:22.787148 30059 log.cc:1196] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-0/wal/wals/c69a720e5f7e4ef68ae42df7535a98cf
I20260929 20:16:22.787477 30061 ts_tablet_manager.cc:1918] T c69a720e5f7e4ef68ae42df7535a98cf P 8e53af7f316f426da3c6262fbd4c43ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:16:22.787633 30061 log.cc:1196] T c69a720e5f7e4ef68ae42df7535a98cf P 8e53af7f316f426da3c6262fbd4c43ca: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-1/wal/wals/c69a720e5f7e4ef68ae42df7535a98cf
I20260929 20:16:22.788183 30061 ts_tablet_manager.cc:1939] T c69a720e5f7e4ef68ae42df7535a98cf P 8e53af7f316f426da3c6262fbd4c43ca: Deleting consensus metadata
I20260929 20:16:22.788178 30059 ts_tablet_manager.cc:1939] T c69a720e5f7e4ef68ae42df7535a98cf P c72beba6466e431fb7bc3bfc3190ec74: Deleting consensus metadata
I20260929 20:16:22.788089 29945 tablet_service.cc:1467] Processing CreateTablet for tablet d4f7b342ca5c4d25a626d8b716e5fc17 (DEFAULT_TABLE table=test-table [id=af78a6cb44eb42bcb254828bb14a3e84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:16:22.788548 29945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4f7b342ca5c4d25a626d8b716e5fc17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:22.791213 30060 ts_tablet_manager.cc:1918] T c69a720e5f7e4ef68ae42df7535a98cf P e10f9c7cfdb84084bc4d25713c4d8325: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:16:22.791410 30060 log.cc:1196] T c69a720e5f7e4ef68ae42df7535a98cf P e10f9c7cfdb84084bc4d25713c4d8325: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-2/wal/wals/c69a720e5f7e4ef68ae42df7535a98cf
I20260929 20:16:22.791802 29509 catalog_manager.cc:4841] TS 8e53af7f316f426da3c6262fbd4c43ca (127.27.185.66:43141): tablet c69a720e5f7e4ef68ae42df7535a98cf (table test-table [id=5cb6a5ea8a404cc09c8fdc7d610b7678]) successfully deleted
I20260929 20:16:22.791934 30060 ts_tablet_manager.cc:1939] T c69a720e5f7e4ef68ae42df7535a98cf P e10f9c7cfdb84084bc4d25713c4d8325: Deleting consensus metadata
I20260929 20:16:22.793156 29512 catalog_manager.cc:4841] TS c72beba6466e431fb7bc3bfc3190ec74 (127.27.185.65:36961): tablet c69a720e5f7e4ef68ae42df7535a98cf (table test-table [id=5cb6a5ea8a404cc09c8fdc7d610b7678]) successfully deleted
I20260929 20:16:22.793962 29509 catalog_manager.cc:4841] TS e10f9c7cfdb84084bc4d25713c4d8325 (127.27.185.67:40393): tablet c69a720e5f7e4ef68ae42df7535a98cf (table test-table [id=5cb6a5ea8a404cc09c8fdc7d610b7678]) successfully deleted
I20260929 20:16:22.797546 30035 tablet_bootstrap.cc:492] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca: Bootstrap starting.
I20260929 20:16:22.798238 30037 tablet_bootstrap.cc:492] T d4f7b342ca5c4d25a626d8b716e5fc17 P e10f9c7cfdb84084bc4d25713c4d8325: Bootstrap starting.
I20260929 20:16:22.799307 30035 tablet_bootstrap.cc:654] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:22.799808 30037 tablet_bootstrap.cc:654] T d4f7b342ca5c4d25a626d8b716e5fc17 P e10f9c7cfdb84084bc4d25713c4d8325: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:22.800541 30036 tablet_bootstrap.cc:492] T d4f7b342ca5c4d25a626d8b716e5fc17 P c72beba6466e431fb7bc3bfc3190ec74: Bootstrap starting.
I20260929 20:16:22.802435 30036 tablet_bootstrap.cc:654] T d4f7b342ca5c4d25a626d8b716e5fc17 P c72beba6466e431fb7bc3bfc3190ec74: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:22.811786 30035 tablet_bootstrap.cc:492] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca: No bootstrap required, opened a new log
I20260929 20:16:22.811985 30035 ts_tablet_manager.cc:1397] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.008s
I20260929 20:16:22.813054 30035 raft_consensus.cc:348] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } }
I20260929 20:16:22.813318 30035 raft_consensus.cc:374] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:22.813419 30035 raft_consensus.cc:729] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e53af7f316f426da3c6262fbd4c43ca, State: Initialized, Role: FOLLOWER
I20260929 20:16:22.813607 30036 tablet_bootstrap.cc:492] T d4f7b342ca5c4d25a626d8b716e5fc17 P c72beba6466e431fb7bc3bfc3190ec74: No bootstrap required, opened a new log
I20260929 20:16:22.813786 30036 ts_tablet_manager.cc:1397] T d4f7b342ca5c4d25a626d8b716e5fc17 P c72beba6466e431fb7bc3bfc3190ec74: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20260929 20:16:22.813766 30035 consensus_queue.cc:260] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } }
I20260929 20:16:22.814230 30037 tablet_bootstrap.cc:492] T d4f7b342ca5c4d25a626d8b716e5fc17 P e10f9c7cfdb84084bc4d25713c4d8325: No bootstrap required, opened a new log
I20260929 20:16:22.814356 30035 ts_tablet_manager.cc:1428] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:16:22.814386 30037 ts_tablet_manager.cc:1397] T d4f7b342ca5c4d25a626d8b716e5fc17 P e10f9c7cfdb84084bc4d25713c4d8325: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.006s
I20260929 20:16:22.814659 30036 raft_consensus.cc:348] T d4f7b342ca5c4d25a626d8b716e5fc17 P c72beba6466e431fb7bc3bfc3190ec74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } }
I20260929 20:16:22.814862 30036 raft_consensus.cc:374] T d4f7b342ca5c4d25a626d8b716e5fc17 P c72beba6466e431fb7bc3bfc3190ec74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:22.814961 30036 raft_consensus.cc:729] T d4f7b342ca5c4d25a626d8b716e5fc17 P c72beba6466e431fb7bc3bfc3190ec74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c72beba6466e431fb7bc3bfc3190ec74, State: Initialized, Role: FOLLOWER
I20260929 20:16:22.815220 30036 consensus_queue.cc:260] T d4f7b342ca5c4d25a626d8b716e5fc17 P c72beba6466e431fb7bc3bfc3190ec74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } }
I20260929 20:16:22.815230 30037 raft_consensus.cc:348] T d4f7b342ca5c4d25a626d8b716e5fc17 P e10f9c7cfdb84084bc4d25713c4d8325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } }
I20260929 20:16:22.815860 30036 ts_tablet_manager.cc:1428] T d4f7b342ca5c4d25a626d8b716e5fc17 P c72beba6466e431fb7bc3bfc3190ec74: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:16:22.815491 30037 raft_consensus.cc:374] T d4f7b342ca5c4d25a626d8b716e5fc17 P e10f9c7cfdb84084bc4d25713c4d8325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:22.816186 30037 raft_consensus.cc:729] T d4f7b342ca5c4d25a626d8b716e5fc17 P e10f9c7cfdb84084bc4d25713c4d8325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e10f9c7cfdb84084bc4d25713c4d8325, State: Initialized, Role: FOLLOWER
I20260929 20:16:22.816474 30037 consensus_queue.cc:260] T d4f7b342ca5c4d25a626d8b716e5fc17 P e10f9c7cfdb84084bc4d25713c4d8325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } }
I20260929 20:16:22.817093 30037 ts_tablet_manager.cc:1428] T d4f7b342ca5c4d25a626d8b716e5fc17 P e10f9c7cfdb84084bc4d25713c4d8325: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 20:16:22.839337 30042 raft_consensus.cc:482] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:16:22.839658 30042 raft_consensus.cc:504] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } }
I20260929 20:16:22.841566 30042 leader_election.cc:288] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c72beba6466e431fb7bc3bfc3190ec74 (127.27.185.65:36961), e10f9c7cfdb84084bc4d25713c4d8325 (127.27.185.67:40393)
I20260929 20:16:22.849665 30066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:22.841955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:40393 (local address 127.27.185.66:51991)
0929 20:16:22.843289 (+ 1334us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:22.843312 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:22.843476 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:22.844013 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:22.844018 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:22.844042 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:16:22.844419 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:22.844430 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.845668 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.845678 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:22.847133 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:22.847145 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.848698 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.848705 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:22.848893 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:22.848931 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 20:16:22.849040 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 20:16:22.849184 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1222,"mutex_wait_us":2,"thread_start_us":745,"threads_started":1}
I20260929 20:16:22.850304 30032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:22.842010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:51947 (local address 127.27.185.65:36961)
0929 20:16:22.842273 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:22.842280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:22.842954 (+ 674us) server_negotiation.cc:407] Connection header received
0929 20:16:22.843300 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:22.843304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:22.843392 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:22.843505 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:16:22.844110 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.845172 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.847754 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.848961 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.849143 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:22.849210 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:22.849815 (+ 605us) server_negotiation.cc:299] Negotiation successful
0929 20:16:22.849959 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260929 20:16:22.849529 30033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:22.842442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:51991 (local address 127.27.185.67:40393)
0929 20:16:22.842620 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:22.842627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:22.843433 (+ 806us) server_negotiation.cc:407] Connection header received
0929 20:16:22.843648 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:22.843654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:22.843732 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:22.843798 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:16:22.844607 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.845535 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.847293 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.848457 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.848637 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:22.848720 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:22.849090 (+ 370us) server_negotiation.cc:299] Negotiation successful
0929 20:16:22.849230 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:16:22.850953 30065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:22.841937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:36961 (local address 127.27.185.66:51947)
0929 20:16:22.842823 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:22.842850 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:22.843090 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:22.843543 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:22.843547 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:22.843575 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:16:22.843942 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:22.843953 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.845326 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.845332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:22.847587 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:22.847602 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.849400 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.849407 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:22.849609 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:22.849642 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:16:22.850495 (+ 853us) client_negotiation.cc:240] Negotiation successful
0929 20:16:22.850638 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":1087,"threads_started":1}
I20260929 20:16:22.850983 29965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4f7b342ca5c4d25a626d8b716e5fc17" candidate_uuid: "8e53af7f316f426da3c6262fbd4c43ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" is_pre_election: true
I20260929 20:16:22.851254 29965 raft_consensus.cc:2454] T d4f7b342ca5c4d25a626d8b716e5fc17 P e10f9c7cfdb84084bc4d25713c4d8325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e53af7f316f426da3c6262fbd4c43ca in term 0.
I20260929 20:16:22.851814 29754 leader_election.cc:302] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e53af7f316f426da3c6262fbd4c43ca, e10f9c7cfdb84084bc4d25713c4d8325; no voters:
I20260929 20:16:22.851879 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4f7b342ca5c4d25a626d8b716e5fc17" candidate_uuid: "8e53af7f316f426da3c6262fbd4c43ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c72beba6466e431fb7bc3bfc3190ec74" is_pre_election: true
I20260929 20:16:22.852285 30042 raft_consensus.cc:2781] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:16:22.852407 30042 raft_consensus.cc:482] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:16:22.852368 29683 raft_consensus.cc:2454] T d4f7b342ca5c4d25a626d8b716e5fc17 P c72beba6466e431fb7bc3bfc3190ec74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e53af7f316f426da3c6262fbd4c43ca in term 0.
I20260929 20:16:22.852494 30042 raft_consensus.cc:3037] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:22.855753 30042 raft_consensus.cc:504] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } }
I20260929 20:16:22.856426 30042 leader_election.cc:288] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca [CANDIDATE]: Term 1 election: Requested vote from peers c72beba6466e431fb7bc3bfc3190ec74 (127.27.185.65:36961), e10f9c7cfdb84084bc4d25713c4d8325 (127.27.185.67:40393)
I20260929 20:16:22.856911 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4f7b342ca5c4d25a626d8b716e5fc17" candidate_uuid: "8e53af7f316f426da3c6262fbd4c43ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c72beba6466e431fb7bc3bfc3190ec74"
I20260929 20:16:22.857012 29965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4f7b342ca5c4d25a626d8b716e5fc17" candidate_uuid: "8e53af7f316f426da3c6262fbd4c43ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e10f9c7cfdb84084bc4d25713c4d8325"
I20260929 20:16:22.857172 29683 raft_consensus.cc:3037] T d4f7b342ca5c4d25a626d8b716e5fc17 P c72beba6466e431fb7bc3bfc3190ec74 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:22.857220 29965 raft_consensus.cc:3037] T d4f7b342ca5c4d25a626d8b716e5fc17 P e10f9c7cfdb84084bc4d25713c4d8325 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:22.860437 29965 raft_consensus.cc:2454] T d4f7b342ca5c4d25a626d8b716e5fc17 P e10f9c7cfdb84084bc4d25713c4d8325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e53af7f316f426da3c6262fbd4c43ca in term 1.
I20260929 20:16:22.860612 29683 raft_consensus.cc:2454] T d4f7b342ca5c4d25a626d8b716e5fc17 P c72beba6466e431fb7bc3bfc3190ec74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e53af7f316f426da3c6262fbd4c43ca in term 1.
I20260929 20:16:22.861104 29754 leader_election.cc:302] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e53af7f316f426da3c6262fbd4c43ca, e10f9c7cfdb84084bc4d25713c4d8325; no voters:
I20260929 20:16:22.861461 30042 raft_consensus.cc:2781] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:16:22.862073 30042 raft_consensus.cc:686] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca [term 1 LEADER]: Becoming Leader. State: Replica: 8e53af7f316f426da3c6262fbd4c43ca, State: Running, Role: LEADER
I20260929 20:16:22.862504 30042 consensus_queue.cc:237] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } }
I20260929 20:16:22.866559 29525 catalog_manager.cc:5495] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e53af7f316f426da3c6262fbd4c43ca (127.27.185.66). New cstate: current_term: 1 leader_uuid: "8e53af7f316f426da3c6262fbd4c43ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } health_report { overall_health: UNKNOWN } } }
Sep 29 20:16:22 dist-test-slave-4cx3 krb5kdc[29479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712980, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:22.925386 30074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:22.906103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39527 (local address 127.0.0.1:41524)
0929 20:16:22.906868 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:22.906890 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:22.907016 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:22.907469 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:22.907472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:22.908165 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:22.908438 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:22.908447 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.909931 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.909934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:22.910904 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:22.910915 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.911946 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.911950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:22.912097 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:22.915938 (+ 3841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:22.915961 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:22.918523 (+ 2562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:22.922462 (+ 3939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:22.922475 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:22.922481 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:22.922917 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:22.923304 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:22.923307 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:22.923309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:22.923499 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:22.924123 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:22.924133 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:22.924355 (+ 222us) client_negotiation.cc:769] Sending connection context
0929 20:16:22.924547 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 20:16:22.924884 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":400,"threads_started":1}
I20260929 20:16:22.925405 29903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:22.905995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41524 (local address 127.27.185.126:39527)
0929 20:16:22.906201 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:22.906210 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:22.906979 (+ 769us) server_negotiation.cc:407] Connection header received
0929 20:16:22.907114 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:22.907120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:22.907237 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:22.907344 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:22.908594 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.909793 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.911045 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:22.911775 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:22.911904 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:22.918753 (+ 6849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:22.918760 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:22.918766 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:22.918807 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:22.922311 (+ 3504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:22.923038 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:22.923044 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:22.923047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:22.923191 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:22.923651 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:22.923661 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:22.923663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:22.923940 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:22.924132 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:22.924674 (+ 542us) server_negotiation.cc:299] Negotiation successful
0929 20:16:22.925029 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":134,"server-negotiator.queue_time_us":68}
I20260929 20:16:22.930819 29525 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:41524:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 20:16:22.932101 29525 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:16:22.939589 29663 tablet_service.cc:1467] Processing CreateTablet for tablet da37224e5281430aa5006b19d95b44c0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ea7a6ed446be48e198e2ffedc8fc3e82]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:16:22.940179 29663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da37224e5281430aa5006b19d95b44c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:22.942548 29804 tablet_service.cc:1467] Processing CreateTablet for tablet da37224e5281430aa5006b19d95b44c0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ea7a6ed446be48e198e2ffedc8fc3e82]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:16:22.943122 29804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da37224e5281430aa5006b19d95b44c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:22.947896 30036 tablet_bootstrap.cc:492] T da37224e5281430aa5006b19d95b44c0 P c72beba6466e431fb7bc3bfc3190ec74: Bootstrap starting.
I20260929 20:16:22.948189 29945 tablet_service.cc:1467] Processing CreateTablet for tablet da37224e5281430aa5006b19d95b44c0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ea7a6ed446be48e198e2ffedc8fc3e82]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:16:22.948786 29945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da37224e5281430aa5006b19d95b44c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:22.949800 30036 tablet_bootstrap.cc:654] T da37224e5281430aa5006b19d95b44c0 P c72beba6466e431fb7bc3bfc3190ec74: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:22.962818 30036 tablet_bootstrap.cc:492] T da37224e5281430aa5006b19d95b44c0 P c72beba6466e431fb7bc3bfc3190ec74: No bootstrap required, opened a new log
I20260929 20:16:22.963035 30036 ts_tablet_manager.cc:1397] T da37224e5281430aa5006b19d95b44c0 P c72beba6466e431fb7bc3bfc3190ec74: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.011s
I20260929 20:16:22.964183 30035 tablet_bootstrap.cc:492] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca: Bootstrap starting.
I20260929 20:16:22.964522 30036 raft_consensus.cc:348] T da37224e5281430aa5006b19d95b44c0 P c72beba6466e431fb7bc3bfc3190ec74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } }
I20260929 20:16:22.964789 30036 raft_consensus.cc:374] T da37224e5281430aa5006b19d95b44c0 P c72beba6466e431fb7bc3bfc3190ec74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:22.964875 30036 raft_consensus.cc:729] T da37224e5281430aa5006b19d95b44c0 P c72beba6466e431fb7bc3bfc3190ec74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c72beba6466e431fb7bc3bfc3190ec74, State: Initialized, Role: FOLLOWER
I20260929 20:16:22.965121 30036 consensus_queue.cc:260] T da37224e5281430aa5006b19d95b44c0 P c72beba6466e431fb7bc3bfc3190ec74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } }
I20260929 20:16:22.965317 30037 tablet_bootstrap.cc:492] T da37224e5281430aa5006b19d95b44c0 P e10f9c7cfdb84084bc4d25713c4d8325: Bootstrap starting.
I20260929 20:16:22.965601 30054 tablet_replica.cc:439] T da37224e5281430aa5006b19d95b44c0 P c72beba6466e431fb7bc3bfc3190ec74: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } } }
I20260929 20:16:22.965947 30054 tablet_replica.cc:442] T da37224e5281430aa5006b19d95b44c0 P c72beba6466e431fb7bc3bfc3190ec74: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:16:22.966017 30035 tablet_bootstrap.cc:654] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:22.966482 30036 ts_tablet_manager.cc:1428] T da37224e5281430aa5006b19d95b44c0 P c72beba6466e431fb7bc3bfc3190ec74: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 20:16:22.967165 30037 tablet_bootstrap.cc:654] T da37224e5281430aa5006b19d95b44c0 P e10f9c7cfdb84084bc4d25713c4d8325: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:22.969163 30035 tablet_bootstrap.cc:492] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca: No bootstrap required, opened a new log
I20260929 20:16:22.969434 30035 ts_tablet_manager.cc:1397] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260929 20:16:22.970541 30035 raft_consensus.cc:348] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } }
I20260929 20:16:22.970870 30035 raft_consensus.cc:374] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:22.970968 30035 raft_consensus.cc:729] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e53af7f316f426da3c6262fbd4c43ca, State: Initialized, Role: FOLLOWER
I20260929 20:16:22.971308 30035 consensus_queue.cc:260] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } }
I20260929 20:16:22.971872 30042 tablet_replica.cc:439] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } } }
I20260929 20:16:22.972155 30042 tablet_replica.cc:442] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:16:22.972687 30035 ts_tablet_manager.cc:1428] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 20:16:22.976423 30037 tablet_bootstrap.cc:492] T da37224e5281430aa5006b19d95b44c0 P e10f9c7cfdb84084bc4d25713c4d8325: No bootstrap required, opened a new log
I20260929 20:16:22.976640 30037 ts_tablet_manager.cc:1397] T da37224e5281430aa5006b19d95b44c0 P e10f9c7cfdb84084bc4d25713c4d8325: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260929 20:16:22.977257 30037 raft_consensus.cc:348] T da37224e5281430aa5006b19d95b44c0 P e10f9c7cfdb84084bc4d25713c4d8325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } }
I20260929 20:16:22.977460 30037 raft_consensus.cc:374] T da37224e5281430aa5006b19d95b44c0 P e10f9c7cfdb84084bc4d25713c4d8325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:22.977545 30037 raft_consensus.cc:729] T da37224e5281430aa5006b19d95b44c0 P e10f9c7cfdb84084bc4d25713c4d8325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e10f9c7cfdb84084bc4d25713c4d8325, State: Initialized, Role: FOLLOWER
I20260929 20:16:22.977820 30037 consensus_queue.cc:260] T da37224e5281430aa5006b19d95b44c0 P e10f9c7cfdb84084bc4d25713c4d8325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } }
I20260929 20:16:22.978381 30037 ts_tablet_manager.cc:1428] T da37224e5281430aa5006b19d95b44c0 P e10f9c7cfdb84084bc4d25713c4d8325: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260929 20:16:22.978314 30043 tablet_replica.cc:439] T da37224e5281430aa5006b19d95b44c0 P e10f9c7cfdb84084bc4d25713c4d8325: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } } }
I20260929 20:16:22.978662 30043 tablet_replica.cc:442] T da37224e5281430aa5006b19d95b44c0 P e10f9c7cfdb84084bc4d25713c4d8325: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:16:23.028565 30042 raft_consensus.cc:482] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:16:23.028796 30042 raft_consensus.cc:504] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } }
I20260929 20:16:23.029464 30042 leader_election.cc:288] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c72beba6466e431fb7bc3bfc3190ec74 (127.27.185.65:36961), e10f9c7cfdb84084bc4d25713c4d8325 (127.27.185.67:40393)
I20260929 20:16:23.030072 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da37224e5281430aa5006b19d95b44c0" candidate_uuid: "8e53af7f316f426da3c6262fbd4c43ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c72beba6466e431fb7bc3bfc3190ec74" is_pre_election: true
I20260929 20:16:23.030102 29965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da37224e5281430aa5006b19d95b44c0" candidate_uuid: "8e53af7f316f426da3c6262fbd4c43ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" is_pre_election: true
I20260929 20:16:23.030391 29683 raft_consensus.cc:2454] T da37224e5281430aa5006b19d95b44c0 P c72beba6466e431fb7bc3bfc3190ec74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e53af7f316f426da3c6262fbd4c43ca in term 0.
I20260929 20:16:23.030433 29965 raft_consensus.cc:2454] T da37224e5281430aa5006b19d95b44c0 P e10f9c7cfdb84084bc4d25713c4d8325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e53af7f316f426da3c6262fbd4c43ca in term 0.
I20260929 20:16:23.030900 29754 leader_election.cc:302] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e53af7f316f426da3c6262fbd4c43ca, e10f9c7cfdb84084bc4d25713c4d8325; no voters:
I20260929 20:16:23.031229 30042 raft_consensus.cc:2781] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:16:23.031337 30042 raft_consensus.cc:482] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:16:23.031517 30042 raft_consensus.cc:3037] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca [term 0 FOLLOWER]: Advancing to term 1
W20260929 20:16:23.033429 29876 tablet.cc:2369] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:16:23.033828 30042 raft_consensus.cc:504] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } }
I20260929 20:16:23.034452 30042 leader_election.cc:288] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca [CANDIDATE]: Term 1 election: Requested vote from peers c72beba6466e431fb7bc3bfc3190ec74 (127.27.185.65:36961), e10f9c7cfdb84084bc4d25713c4d8325 (127.27.185.67:40393)
I20260929 20:16:23.034976 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da37224e5281430aa5006b19d95b44c0" candidate_uuid: "8e53af7f316f426da3c6262fbd4c43ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c72beba6466e431fb7bc3bfc3190ec74"
I20260929 20:16:23.035002 29965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da37224e5281430aa5006b19d95b44c0" candidate_uuid: "8e53af7f316f426da3c6262fbd4c43ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e10f9c7cfdb84084bc4d25713c4d8325"
I20260929 20:16:23.035230 29683 raft_consensus.cc:3037] T da37224e5281430aa5006b19d95b44c0 P c72beba6466e431fb7bc3bfc3190ec74 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:23.035247 29965 raft_consensus.cc:3037] T da37224e5281430aa5006b19d95b44c0 P e10f9c7cfdb84084bc4d25713c4d8325 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:23.037496 29683 raft_consensus.cc:2454] T da37224e5281430aa5006b19d95b44c0 P c72beba6466e431fb7bc3bfc3190ec74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e53af7f316f426da3c6262fbd4c43ca in term 1.
I20260929 20:16:23.038074 29757 leader_election.cc:302] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e53af7f316f426da3c6262fbd4c43ca, c72beba6466e431fb7bc3bfc3190ec74; no voters:
I20260929 20:16:23.038266 29965 raft_consensus.cc:2454] T da37224e5281430aa5006b19d95b44c0 P e10f9c7cfdb84084bc4d25713c4d8325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e53af7f316f426da3c6262fbd4c43ca in term 1.
I20260929 20:16:23.038398 30042 raft_consensus.cc:2781] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:16:23.038576 30042 raft_consensus.cc:686] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca [term 1 LEADER]: Becoming Leader. State: Replica: 8e53af7f316f426da3c6262fbd4c43ca, State: Running, Role: LEADER
I20260929 20:16:23.038844 30042 consensus_queue.cc:237] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } }
I20260929 20:16:23.041939 30067 tablet_replica.cc:439] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca: TxnStatusTablet state changed. Reason: New leader 8e53af7f316f426da3c6262fbd4c43ca. Latest consensus state: current_term: 1 leader_uuid: "8e53af7f316f426da3c6262fbd4c43ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } } }
I20260929 20:16:23.042363 30067 tablet_replica.cc:442] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:16:23.043170 30079 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 20:16:23.043447 30079 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 20:16:23.044065 29523 catalog_manager.cc:5495] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e53af7f316f426da3c6262fbd4c43ca (127.27.185.66). New cstate: current_term: 1 leader_uuid: "8e53af7f316f426da3c6262fbd4c43ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } health_report { overall_health: UNKNOWN } } }
I20260929 20:16:23.077350 30031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:23.071461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48312 (local address 127.27.185.66:43141)
0929 20:16:23.071660 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:23.071667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:23.071887 (+ 220us) server_negotiation.cc:407] Connection header received
0929 20:16:23.072213 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:23.072219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:23.072342 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:23.072510 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:16:23.072877 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:23.073682 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:23.074991 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:23.075512 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:23.075656 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:23.076114 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:16:23.076666 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:16:23.076822 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:23.076901 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 20:16:23.076984 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:16:23.077323 30074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:23.071589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:43141 (local address 127.0.0.1:48312)
0929 20:16:23.071794 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:23.071810 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:23.071974 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:23.072535 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:23.072539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:23.072551 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:16:23.072742 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:23.072750 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:23.073820 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:23.073825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:23.074843 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:23.074854 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:23.075664 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:23.075670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:23.075857 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:23.075913 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:16:23.076768 (+ 855us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:16:23.076791 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:16:23.076882 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 20:16:23.077038 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 20:16:23.081607 29965 raft_consensus.cc:1261] T da37224e5281430aa5006b19d95b44c0 P e10f9c7cfdb84084bc4d25713c4d8325 [term 1 FOLLOWER]: Refusing update from remote peer 8e53af7f316f426da3c6262fbd4c43ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:16:23.081969 29683 raft_consensus.cc:1261] T da37224e5281430aa5006b19d95b44c0 P c72beba6466e431fb7bc3bfc3190ec74 [term 1 FOLLOWER]: Refusing update from remote peer 8e53af7f316f426da3c6262fbd4c43ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:16:23.082420 30042 consensus_queue.cc:1035] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:16:23.083472 30042 consensus_queue.cc:1035] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:16:23.085114 30043 tablet_replica.cc:439] T da37224e5281430aa5006b19d95b44c0 P e10f9c7cfdb84084bc4d25713c4d8325: TxnStatusTablet state changed. Reason: New leader 8e53af7f316f426da3c6262fbd4c43ca. Latest consensus state: current_term: 1 leader_uuid: "8e53af7f316f426da3c6262fbd4c43ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } } }
I20260929 20:16:23.085376 30043 tablet_replica.cc:442] T da37224e5281430aa5006b19d95b44c0 P e10f9c7cfdb84084bc4d25713c4d8325: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:16:23.087591 30067 tablet_replica.cc:439] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8e53af7f316f426da3c6262fbd4c43ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } } }
I20260929 20:16:23.087818 30067 tablet_replica.cc:442] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:16:23.088681 30043 tablet_replica.cc:439] T da37224e5281430aa5006b19d95b44c0 P e10f9c7cfdb84084bc4d25713c4d8325: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8e53af7f316f426da3c6262fbd4c43ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } } }
I20260929 20:16:23.088896 30043 tablet_replica.cc:442] T da37224e5281430aa5006b19d95b44c0 P e10f9c7cfdb84084bc4d25713c4d8325: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:16:23.089656 30051 mvcc.cc:205] Tried to move back new op lower bound from 7334760378694713344 to 7334760378530205696. Current Snapshot: MvccSnapshot[applied={T|T < 7334760378694713344 or (T in {7334760378694713344})}]
I20260929 20:16:23.098934 30054 tablet_replica.cc:439] T da37224e5281430aa5006b19d95b44c0 P c72beba6466e431fb7bc3bfc3190ec74: TxnStatusTablet state changed. Reason: New leader 8e53af7f316f426da3c6262fbd4c43ca. Latest consensus state: current_term: 1 leader_uuid: "8e53af7f316f426da3c6262fbd4c43ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } } }
I20260929 20:16:23.099267 30054 tablet_replica.cc:442] T da37224e5281430aa5006b19d95b44c0 P c72beba6466e431fb7bc3bfc3190ec74: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:16:23.104020 30067 tablet_replica.cc:439] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8e53af7f316f426da3c6262fbd4c43ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } } }
I20260929 20:16:23.104285 30067 tablet_replica.cc:442] T da37224e5281430aa5006b19d95b44c0 P 8e53af7f316f426da3c6262fbd4c43ca: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:16:23.105914 30031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:23.099546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48322 (local address 127.27.185.66:43141)
0929 20:16:23.099736 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:23.099742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:23.099760 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:16:23.099822 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:23.099827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:23.099927 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:23.100021 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:16:23.100614 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:23.101779 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:23.103149 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:23.103802 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:23.103926 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:23.104615 (+ 689us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:16:23.104756 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:16:23.104842 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:23.105449 (+ 607us) server_negotiation.cc:299] Negotiation successful
0929 20:16:23.105568 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:16:23.108567 30054 tablet_replica.cc:439] T da37224e5281430aa5006b19d95b44c0 P c72beba6466e431fb7bc3bfc3190ec74: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8e53af7f316f426da3c6262fbd4c43ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 } } peers { permanent_uuid: "8e53af7f316f426da3c6262fbd4c43ca" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43141 } } peers { permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 } } }
I20260929 20:16:23.108804 30054 tablet_replica.cc:442] T da37224e5281430aa5006b19d95b44c0 P c72beba6466e431fb7bc3bfc3190ec74: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:16:23.128832 30031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:23.118664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:40367 (local address 127.27.185.66:43141)
0929 20:16:23.118922 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:23.118932 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:23.123084 (+ 4152us) server_negotiation.cc:407] Connection header received
0929 20:16:23.123353 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:23.123358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:23.123477 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:23.123561 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:16:23.124203 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:23.125246 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:23.126853 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:23.127333 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:23.127550 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:23.127981 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:16:23.128108 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:16:23.128225 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:23.128386 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 20:16:23.128463 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:16:23.129333 30086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:23.118626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:43141 (local address 127.27.185.66:40367)
0929 20:16:23.122973 (+ 4347us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:23.122991 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:23.123184 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:23.123585 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:23.123589 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:23.123601 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:16:23.124051 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:23.124060 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:23.125415 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:23.125421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:23.126674 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:23.126686 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:23.127570 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:23.127576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:23.127734 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:23.127790 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:16:23.128221 (+ 431us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:16:23.128243 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 20:16:23.128360 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 20:16:23.128505 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4173,"thread_start_us":3985,"threads_started":1}
I20260929 20:16:23.141561 29965 raft_consensus.cc:1261] T d4f7b342ca5c4d25a626d8b716e5fc17 P e10f9c7cfdb84084bc4d25713c4d8325 [term 1 FOLLOWER]: Refusing update from remote peer 8e53af7f316f426da3c6262fbd4c43ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:16:23.141767 29682 raft_consensus.cc:1261] T d4f7b342ca5c4d25a626d8b716e5fc17 P c72beba6466e431fb7bc3bfc3190ec74 [term 1 FOLLOWER]: Refusing update from remote peer 8e53af7f316f426da3c6262fbd4c43ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:16:23.142241 30042 consensus_queue.cc:1035] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "e10f9c7cfdb84084bc4d25713c4d8325" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:16:23.142680 30067 consensus_queue.cc:1035] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "c72beba6466e431fb7bc3bfc3190ec74" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:16:23.238597 29525 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:16:23.238888 29525 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:16:23.242113 29525 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2739ae47e0fd4fa49075618456af760d: Sending DeleteTablet for 3 replicas of tablet d4f7b342ca5c4d25a626d8b716e5fc17
I20260929 20:16:23.243162 29945 tablet_service.cc:1514] Processing DeleteTablet for tablet d4f7b342ca5c4d25a626d8b716e5fc17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:16:23 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:51080
I20260929 20:16:23.243302 29803 tablet_service.cc:1514] Processing DeleteTablet for tablet d4f7b342ca5c4d25a626d8b716e5fc17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:16:23 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:48298
I20260929 20:16:23.243558 30060 tablet_replica.cc:330] T d4f7b342ca5c4d25a626d8b716e5fc17 P e10f9c7cfdb84084bc4d25713c4d8325: stopping tablet replica
I20260929 20:16:23.243623 30061 tablet_replica.cc:330] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca: stopping tablet replica
I20260929 20:16:23.243794 30060 raft_consensus.cc:2229] T d4f7b342ca5c4d25a626d8b716e5fc17 P e10f9c7cfdb84084bc4d25713c4d8325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:16:23.243847 30061 raft_consensus.cc:2229] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:16:23.244026 30060 raft_consensus.cc:2258] T d4f7b342ca5c4d25a626d8b716e5fc17 P e10f9c7cfdb84084bc4d25713c4d8325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:16:23.244379 30061 raft_consensus.cc:2258] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:16:23.245280 30060 ts_tablet_manager.cc:1905] T d4f7b342ca5c4d25a626d8b716e5fc17 P e10f9c7cfdb84084bc4d25713c4d8325: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:16:23.245625 29663 tablet_service.cc:1514] Processing DeleteTablet for tablet d4f7b342ca5c4d25a626d8b716e5fc17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:16:23 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:50024
I20260929 20:16:23.245940 30059 tablet_replica.cc:330] T d4f7b342ca5c4d25a626d8b716e5fc17 P c72beba6466e431fb7bc3bfc3190ec74: stopping tablet replica
I20260929 20:16:23.246140 30059 raft_consensus.cc:2229] T d4f7b342ca5c4d25a626d8b716e5fc17 P c72beba6466e431fb7bc3bfc3190ec74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:16:23.246330 30059 raft_consensus.cc:2258] T d4f7b342ca5c4d25a626d8b716e5fc17 P c72beba6466e431fb7bc3bfc3190ec74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:16:23.247524 30059 ts_tablet_manager.cc:1905] T d4f7b342ca5c4d25a626d8b716e5fc17 P c72beba6466e431fb7bc3bfc3190ec74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:16:23.252310 30061 ts_tablet_manager.cc:1905] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:16:23.254377 30059 ts_tablet_manager.cc:1918] T d4f7b342ca5c4d25a626d8b716e5fc17 P c72beba6466e431fb7bc3bfc3190ec74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:16:23.254531 30059 log.cc:1196] T d4f7b342ca5c4d25a626d8b716e5fc17 P c72beba6466e431fb7bc3bfc3190ec74: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-0/wal/wals/d4f7b342ca5c4d25a626d8b716e5fc17
I20260929 20:16:23.254990 30059 ts_tablet_manager.cc:1939] T d4f7b342ca5c4d25a626d8b716e5fc17 P c72beba6466e431fb7bc3bfc3190ec74: Deleting consensus metadata
I20260929 20:16:23.265050 30061 ts_tablet_manager.cc:1918] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:16:23.265264 30061 log.cc:1196] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-1/wal/wals/d4f7b342ca5c4d25a626d8b716e5fc17
I20260929 20:16:23.265884 30061 ts_tablet_manager.cc:1939] T d4f7b342ca5c4d25a626d8b716e5fc17 P 8e53af7f316f426da3c6262fbd4c43ca: Deleting consensus metadata
I20260929 20:16:23.268239 29512 catalog_manager.cc:4841] TS c72beba6466e431fb7bc3bfc3190ec74 (127.27.185.65:36961): tablet d4f7b342ca5c4d25a626d8b716e5fc17 (table test-table [id=af78a6cb44eb42bcb254828bb14a3e84]) successfully deleted
I20260929 20:16:23.270627 30060 ts_tablet_manager.cc:1918] T d4f7b342ca5c4d25a626d8b716e5fc17 P e10f9c7cfdb84084bc4d25713c4d8325: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:16:23.270849 30060 log.cc:1196] T d4f7b342ca5c4d25a626d8b716e5fc17 P e10f9c7cfdb84084bc4d25713c4d8325: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790712966917858-28389-0/minicluster-data/ts-2/wal/wals/d4f7b342ca5c4d25a626d8b716e5fc17
I20260929 20:16:23.272409 29509 catalog_manager.cc:4841] TS 8e53af7f316f426da3c6262fbd4c43ca (127.27.185.66:43141): tablet d4f7b342ca5c4d25a626d8b716e5fc17 (table test-table [id=af78a6cb44eb42bcb254828bb14a3e84]) successfully deleted
I20260929 20:16:23.272713 30060 ts_tablet_manager.cc:1939] T d4f7b342ca5c4d25a626d8b716e5fc17 P e10f9c7cfdb84084bc4d25713c4d8325: Deleting consensus metadata
I20260929 20:16:23.273957 29509 catalog_manager.cc:4841] TS e10f9c7cfdb84084bc4d25713c4d8325 (127.27.185.67:40393): tablet d4f7b342ca5c4d25a626d8b716e5fc17 (table test-table [id=af78a6cb44eb42bcb254828bb14a3e84]) successfully deleted
Sep 29 20:16:23 dist-test-slave-4cx3 krb5kdc[29479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712982, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.185.65@KRBTEST.COM
I20260929 20:16:23.289469 30032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:23.257942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50038 (local address 127.27.185.65:36961)
0929 20:16:23.258173 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:23.258179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:23.272110 (+ 13931us) server_negotiation.cc:407] Connection header received
0929 20:16:23.272336 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:23.272342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:23.272499 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:23.272627 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:23.275022 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:23.275970 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:23.277374 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:23.277865 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:23.278014 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:23.282490 (+ 4476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:23.282504 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:23.282521 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:23.282561 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:23.286009 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:23.286778 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:23.286789 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:23.286793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:23.286974 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:23.287560 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:23.287568 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:23.287572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:23.288098 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:23.288617 (+ 519us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:23.288815 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 20:16:23.289097 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 20:16:23.290445 29729 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50038
I20260929 20:16:23.314248 29903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:23.293866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41528 (local address 127.27.185.126:39527)
0929 20:16:23.294075 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:23.294082 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:23.300758 (+ 6676us) server_negotiation.cc:407] Connection header received
0929 20:16:23.301063 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:23.301068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:23.301141 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:23.301204 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:23.302688 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:23.303543 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:23.304836 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:23.305411 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:23.305615 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:23.308525 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:23.308530 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:23.308534 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:23.308568 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:23.311264 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:23.312044 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:23.312054 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:23.312056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:23.312296 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:23.312724 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:23.312728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:23.312731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:23.313013 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:23.313272 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:23.313632 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 20:16:23.313859 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 20:16:23.314832 29525 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41528
I20260929 20:16:23.316987 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 29597
I20260929 20:16:23.343153 30032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:23.318219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50044 (local address 127.27.185.65:36961)
0929 20:16:23.320202 (+ 1983us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:23.320211 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:23.323358 (+ 3147us) server_negotiation.cc:407] Connection header received
0929 20:16:23.323585 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:23.323591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:23.323666 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:23.323774 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:23.325055 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:23.326121 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:23.332363 (+ 6242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:23.333025 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:23.333121 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:23.336218 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:23.336225 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:23.336230 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:23.336274 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:23.339951 (+ 3677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:23.340782 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:23.340788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:23.340792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:23.340969 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:23.341382 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:23.341387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:23.341391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:23.341725 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:23.341961 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:23.342544 (+ 583us) server_negotiation.cc:299] Negotiation successful
0929 20:16:23.342776 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1816}
W20260929 20:16:23.344725 29729 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50044
W20260929 20:16:23.345275 28389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 29597: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:16:23.345500 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 29597
I20260929 20:16:23.375478 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 29738
Sep 29 20:16:23 dist-test-slave-4cx3 krb5kdc[29479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712982, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.185.66@KRBTEST.COM
I20260929 20:16:23.395067 30031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:23.376253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48332 (local address 127.27.185.66:43141)
0929 20:16:23.376451 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:23.376457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:23.376530 (+ 73us) server_negotiation.cc:407] Connection header received
0929 20:16:23.376696 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:23.376702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:23.376784 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:23.376894 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:23.378379 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:23.379546 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:23.381260 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:23.381980 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:23.382119 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:23.387559 (+ 5440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:23.387572 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:23.387591 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:23.387656 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:23.391698 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:23.392580 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:23.392589 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:23.392593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:23.392757 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:23.393211 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:23.393216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:23.393220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:23.393753 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:23.394051 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:23.394380 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 20:16:23.394654 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 20:16:23.395921 29870 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48332
W20260929 20:16:23.396732 28389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 29738: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:16:23.396898 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 29738
I20260929 20:16:23.416460 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 29879
Sep 29 20:16:23 dist-test-slave-4cx3 krb5kdc[29479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712982, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.185.67@KRBTEST.COM
I20260929 20:16:23.432633 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:23.417175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51088 (local address 127.27.185.67:40393)
0929 20:16:23.417679 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:23.417684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:23.417698 (+ 14us) server_negotiation.cc:407] Connection header received
0929 20:16:23.417751 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:23.417755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:23.417812 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:23.417939 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:23.419231 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:23.419991 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:23.421185 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:23.421644 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:23.421784 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:23.426649 (+ 4865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:23.426665 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:23.426681 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:23.426720 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:23.430034 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:23.430725 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:23.430732 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:23.430735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:23.430867 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:23.431189 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:23.431192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:23.431194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:23.431750 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:23.431981 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:23.432181 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 20:16:23.432356 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":249,"threads_started":1}
W20260929 20:16:23.433338 30011 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51088
W20260929 20:16:23.434031 28389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 29879: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:16:23.434190 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 29879
I20260929 20:16:23.451489 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 29493
I20260929 20:16:23.452674 29565 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:16:23.570737 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 29493
2026-09-29T20:16:23Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4937 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:16:23 dist-test-slave-4cx3 krb5kdc[30100](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:16:23 dist-test-slave-4cx3 krb5kdc[30100](info): set up 2 sockets
Sep 29 20:16:23 dist-test-slave-4cx3 krb5kdc[30100](info): commencing operation
krb5kdc: starting...
W20260929 20:16:24.662635 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:16:24 dist-test-slave-4cx3 krb5kdc[30100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:16:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:16:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
I20260929 20:16:24.851799 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:45305
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:40559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:45305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:25.169476 30113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:25.169770 30113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:25.169936 30113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:25.177212 30113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:16:25.177352 30113 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 20:16:25.177421 30113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:25.177536 30113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:25.177621 30113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:16:25.177721 30113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:16:25.187788 30113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:40559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:45305
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:45305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:25.189114 30113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:25.190549 30113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:25.202716 30118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:25.202736 30119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:25.203142 30121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:25.203711 30113 server_base.cc:1034] running on GCE node
I20260929 20:16:25.204535 30113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:25.206991 30113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:25.208228 30113 hybrid_clock.cc:648] HybridClock initialized: now 1790712985208158 us; error 49 us; skew 500 ppm
I20260929 20:16:25.208575 30113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:25 dist-test-slave-4cx3 krb5kdc[30100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712985, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:25.211870 30113 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:16:25.213449 30113 webserver.cc:460] Webserver started at http://127.27.185.126:36047/ using document root <none> and password file <none>
I20260929 20:16:25.214171 30113 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:25.214309 30113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:25.214630 30113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:25.217659 30113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "34f1fba2fa284ab08e77a76867895f77"
format_stamp: "Formatted at 2026-09-29 20:16:25 on dist-test-slave-4cx3"
server_key: "e340c8e1513e80e0cc95435c85624af0"
server_key_iv: "5aea178515ff95691852ff27ee44acdf"
server_key_version: "encryptionkey@0"
I20260929 20:16:25.218500 30113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "34f1fba2fa284ab08e77a76867895f77"
format_stamp: "Formatted at 2026-09-29 20:16:25 on dist-test-slave-4cx3"
server_key: "e340c8e1513e80e0cc95435c85624af0"
server_key_iv: "5aea178515ff95691852ff27ee44acdf"
server_key_version: "encryptionkey@0"
I20260929 20:16:25.223836 30113 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:16:25.227680 30128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:25.229115 30113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:16:25.229344 30113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "34f1fba2fa284ab08e77a76867895f77"
format_stamp: "Formatted at 2026-09-29 20:16:25 on dist-test-slave-4cx3"
server_key: "e340c8e1513e80e0cc95435c85624af0"
server_key_iv: "5aea178515ff95691852ff27ee44acdf"
server_key_version: "encryptionkey@0"
I20260929 20:16:25.229570 30113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:25.254209 30113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:25.258747 30113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:25.259044 30113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:25.279745 30113 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:45305
I20260929 20:16:25.279765 30190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:45305 every 8 connection(s)
I20260929 20:16:25.281136 30113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:16:25.285009 30191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:25.289331 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 30113
I20260929 20:16:25.289552 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:16:25.290284 28389 external_mini_cluster.cc:1428] Setting key c96ae2cb7b14aacae6bf6976af4860da
I20260929 20:16:25.295573 30191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34f1fba2fa284ab08e77a76867895f77: Bootstrap starting.
I20260929 20:16:25.299227 30191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34f1fba2fa284ab08e77a76867895f77: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:25.300549 30191 log.cc:824] T 00000000000000000000000000000000 P 34f1fba2fa284ab08e77a76867895f77: Log is configured to *not* fsync() on all Append() calls
I20260929 20:16:25.304318 30191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34f1fba2fa284ab08e77a76867895f77: No bootstrap required, opened a new log
Sep 29 20:16:25 dist-test-slave-4cx3 krb5kdc[30100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:25.312641 30191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34f1fba2fa284ab08e77a76867895f77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f1fba2fa284ab08e77a76867895f77" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 45305 } }
I20260929 20:16:25.313076 30191 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34f1fba2fa284ab08e77a76867895f77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:25.313169 30191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34f1fba2fa284ab08e77a76867895f77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34f1fba2fa284ab08e77a76867895f77, State: Initialized, Role: FOLLOWER
I20260929 20:16:25.313858 30191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34f1fba2fa284ab08e77a76867895f77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f1fba2fa284ab08e77a76867895f77" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 45305 } }
I20260929 20:16:25.314097 30191 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34f1fba2fa284ab08e77a76867895f77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:16:25.314242 30191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34f1fba2fa284ab08e77a76867895f77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:16:25.314347 30194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:25.292331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55020 (local address 127.27.185.126:45305)
0929 20:16:25.293156 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:25.293168 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:25.293214 (+ 46us) server_negotiation.cc:407] Connection header received
0929 20:16:25.294886 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:25.294909 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:25.295293 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:25.295699 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:25.297498 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:25.298871 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:25.300735 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:25.301862 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:25.302041 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:25.306682 (+ 4641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:25.306690 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:25.306708 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:25.306747 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:25.309885 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:25.310603 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:25.310610 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:25.310617 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:25.310816 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:25.311240 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:25.311244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:25.311246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:25.311867 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:25.312175 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:25.312618 (+ 443us) server_negotiation.cc:299] Negotiation successful
0929 20:16:25.313112 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":386,"threads_started":1}
I20260929 20:16:25.314435 30191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34f1fba2fa284ab08e77a76867895f77 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:25.316829 30191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34f1fba2fa284ab08e77a76867895f77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f1fba2fa284ab08e77a76867895f77" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 45305 } }
I20260929 20:16:25.317391 30191 leader_election.cc:302] T 00000000000000000000000000000000 P 34f1fba2fa284ab08e77a76867895f77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34f1fba2fa284ab08e77a76867895f77; no voters:
I20260929 20:16:25.317931 30191 leader_election.cc:288] T 00000000000000000000000000000000 P 34f1fba2fa284ab08e77a76867895f77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:16:25.318085 30196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34f1fba2fa284ab08e77a76867895f77 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:16:25.318885 30196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34f1fba2fa284ab08e77a76867895f77 [term 1 LEADER]: Becoming Leader. State: Replica: 34f1fba2fa284ab08e77a76867895f77, State: Running, Role: LEADER
I20260929 20:16:25.319543 30196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34f1fba2fa284ab08e77a76867895f77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f1fba2fa284ab08e77a76867895f77" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 45305 } }
I20260929 20:16:25.320139 30191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34f1fba2fa284ab08e77a76867895f77 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:16:25.322798 30197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34f1fba2fa284ab08e77a76867895f77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34f1fba2fa284ab08e77a76867895f77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f1fba2fa284ab08e77a76867895f77" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 45305 } } }
I20260929 20:16:25.323325 30197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34f1fba2fa284ab08e77a76867895f77 [sys.catalog]: This master's current role is: LEADER
I20260929 20:16:25.323156 30198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34f1fba2fa284ab08e77a76867895f77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34f1fba2fa284ab08e77a76867895f77. Latest consensus state: current_term: 1 leader_uuid: "34f1fba2fa284ab08e77a76867895f77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f1fba2fa284ab08e77a76867895f77" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 45305 } } }
I20260929 20:16:25.323553 30198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34f1fba2fa284ab08e77a76867895f77 [sys.catalog]: This master's current role is: LEADER
I20260929 20:16:25.336326 30206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:16:25.341629 30206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:16:25.350553 30206 catalog_manager.cc:1347] Generated new cluster ID: 03da62f091684f1c868c4a24959ee009
I20260929 20:16:25.350678 30206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:16:25.400154 30206 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:16:25.401495 30206 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:16:25.409652 30206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34f1fba2fa284ab08e77a76867895f77: Generated new TSK 0
I20260929 20:16:25.410480 30206 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
I20260929 20:16:25.460985 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:0
--local_ip_for_outbound_sockets=127.27.185.65
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:45305
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:40559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:25.824666 30218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:25.824963 30218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:25.825070 30218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:25.832368 30218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:25.832481 30218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:25.832576 30218 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 20:16:25.832628 30218 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:16:25.832690 30218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:16:25.842864 30218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:40559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.185.126:45305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:25.844213 30218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:25.845528 30218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:25.858487 30223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:25.858596 30224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:25.858660 30226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:25.859369 30218 server_base.cc:1034] running on GCE node
I20260929 20:16:25.859994 30218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:25.860886 30218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:25.862093 30218 hybrid_clock.cc:648] HybridClock initialized: now 1790712985862057 us; error 33 us; skew 500 ppm
I20260929 20:16:25.862378 30218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:25 dist-test-slave-4cx3 krb5kdc[30100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712985, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:25.865556 30218 init.cc:377] Logged in from keytab as kudu/127.27.185.65@KRBTEST.COM (short username kudu)
I20260929 20:16:25.867065 30218 webserver.cc:460] Webserver started at http://127.27.185.65:32955/ using document root <none> and password file <none>
I20260929 20:16:25.867841 30218 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:25.867997 30218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:25.868337 30218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:25.871295 30218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-0/data/instance:
uuid: "058bb4d2a4d8493c8a15a926674d3a73"
format_stamp: "Formatted at 2026-09-29 20:16:25 on dist-test-slave-4cx3"
server_key: "1d2a93ebf7f5d544224e373b147211b3"
server_key_iv: "d940df8ccaf2fafe0027414ab58bfed2"
server_key_version: "encryptionkey@0"
I20260929 20:16:25.872118 30218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance:
uuid: "058bb4d2a4d8493c8a15a926674d3a73"
format_stamp: "Formatted at 2026-09-29 20:16:25 on dist-test-slave-4cx3"
server_key: "1d2a93ebf7f5d544224e373b147211b3"
server_key_iv: "d940df8ccaf2fafe0027414ab58bfed2"
server_key_version: "encryptionkey@0"
I20260929 20:16:25.877255 30218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:16:25.880750 30233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:25.882037 30218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:16:25.882217 30218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "058bb4d2a4d8493c8a15a926674d3a73"
format_stamp: "Formatted at 2026-09-29 20:16:25 on dist-test-slave-4cx3"
server_key: "1d2a93ebf7f5d544224e373b147211b3"
server_key_iv: "d940df8ccaf2fafe0027414ab58bfed2"
server_key_version: "encryptionkey@0"
I20260929 20:16:25.882421 30218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:25.907917 30218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:25.912225 30218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:25.912523 30218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:25.915146 30218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:16:25.915282 30218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:25.915436 30218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:16:25.915546 30218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 29 20:16:25 dist-test-slave-4cx3 krb5kdc[30100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712985, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:25.941188 30242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:25.915869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:35121 (local address 127.27.185.126:45305)
0929 20:16:25.918194 (+ 2325us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:25.918200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:25.918223 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:16:25.919827 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:25.919836 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:25.919942 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:25.920082 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:25.922387 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:25.923854 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:25.927174 (+ 3320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:25.927785 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:25.927891 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:25.934138 (+ 6247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:25.934147 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:25.934153 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:25.934206 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:25.937123 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:25.938338 (+ 1215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:25.938345 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:25.938350 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:25.938515 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:25.939181 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:25.939188 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:25.939194 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:25.939687 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:25.939836 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:25.940581 (+ 745us) server_negotiation.cc:299] Negotiation successful
0929 20:16:25.940803 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2198,"thread_start_us":2066,"threads_started":1}
I20260929 20:16:25.942279 30243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:25.916489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:45305 (local address 127.27.185.65:35121)
0929 20:16:25.917351 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:25.917398 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:25.919427 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:25.920374 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:25.920390 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:25.921234 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:25.922156 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:25.922191 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:25.924089 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:25.924098 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:25.927012 (+ 2914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:25.927029 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:25.928079 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:25.928089 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:25.928451 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:25.929341 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:25.929386 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:25.933757 (+ 4371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:25.937285 (+ 3528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:25.937304 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:25.937328 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:25.938192 (+ 864us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:25.938668 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:25.938675 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:25.938686 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:25.939032 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:25.939836 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:25.939849 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:25.940202 (+ 353us) client_negotiation.cc:769] Sending connection context
0929 20:16:25.940478 (+ 276us) client_negotiation.cc:240] Negotiation successful
0929 20:16:25.941047 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":363,"threads_started":1}
I20260929 20:16:25.962644 30218 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:38361
I20260929 20:16:25.962680 30355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:38361 every 8 connection(s)
I20260929 20:16:25.964356 30218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:16:25.966886 30356 heartbeater.cc:347] Connected to a master server at 127.27.185.126:45305
I20260929 20:16:25.967139 30356 heartbeater.cc:464] Registering TS with master...
I20260929 20:16:25.967973 30356 heartbeater.cc:511] Master 127.27.185.126:45305 requested a full tablet report, sending...
I20260929 20:16:25.970257 30145 ts_manager.cc:194] Registered new tserver with Master: 058bb4d2a4d8493c8a15a926674d3a73 (127.27.185.65:38361)
I20260929 20:16:25.970680 30242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:25.950186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:44797 (local address 127.27.185.126:45305)
0929 20:16:25.950420 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:25.950429 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:25.956138 (+ 5709us) server_negotiation.cc:407] Connection header received
0929 20:16:25.956337 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:25.956340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:25.956402 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:25.956556 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:25.957933 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:25.959207 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:25.961542 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:25.962118 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:25.962264 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:25.964515 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:25.964522 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:25.964531 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:25.964562 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:25.967693 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:25.968592 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:25.968606 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:25.968609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:25.968777 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:25.969198 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:25.969203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:25.969207 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:25.969523 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:25.969713 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:25.970079 (+ 366us) server_negotiation.cc:299] Negotiation successful
0929 20:16:25.970306 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:16:25.970741 30322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:25.950158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:45305 (local address 127.27.185.65:44797)
0929 20:16:25.955993 (+ 5835us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:25.956030 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:25.956213 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:25.956589 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:25.956595 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:25.957250 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:25.957729 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:25.957747 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:25.959441 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:25.959449 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:25.961391 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:25.961408 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:25.962341 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:25.962351 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:25.962582 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:25.963222 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:25.963244 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:25.964289 (+ 1045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:25.967836 (+ 3547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:25.967841 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:25.967845 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:25.968421 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:25.968939 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:25.968942 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:25.968945 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:25.969071 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:25.969645 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:25.969650 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:25.969835 (+ 185us) client_negotiation.cc:769] Sending connection context
0929 20:16:25.969972 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 20:16:25.970298 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5633,"thread_start_us":5469,"threads_started":1}
I20260929 20:16:25.972288 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 30218
I20260929 20:16:25.972540 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance
I20260929 20:16:25.972549 30145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.65@KRBTEST.COM'} at 127.27.185.65:35121
I20260929 20:16:25.973364 28389 external_mini_cluster.cc:1428] Setting key 3700b9c1dddfff6e08641d113e583b99
WARNING: no policy specified for kudu/127.27.185.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
I20260929 20:16:26.014845 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:0
--local_ip_for_outbound_sockets=127.27.185.66
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:45305
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:40559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:26.345705 30360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:26.346101 30360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:26.346233 30360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:26.353449 30360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:26.353569 30360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:26.353657 30360 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 20:16:26.353709 30360 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:16:26.353773 30360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:16:26.363730 30360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:40559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.185.126:45305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:26.365025 30360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:26.366289 30360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:26.380228 30365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:26.380254 30368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:26.380228 30366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:26.381568 30360 server_base.cc:1034] running on GCE node
I20260929 20:16:26.382223 30360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:26.383215 30360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:26.384426 30360 hybrid_clock.cc:648] HybridClock initialized: now 1790712986384394 us; error 101 us; skew 500 ppm
I20260929 20:16:26.384714 30360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:26 dist-test-slave-4cx3 krb5kdc[30100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712986, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:26.388176 30360 init.cc:377] Logged in from keytab as kudu/127.27.185.66@KRBTEST.COM (short username kudu)
I20260929 20:16:26.389605 30360 webserver.cc:460] Webserver started at http://127.27.185.66:38623/ using document root <none> and password file <none>
I20260929 20:16:26.390259 30360 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:26.390357 30360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:26.390617 30360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:26.393512 30360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-1/data/instance:
uuid: "51c98447074e4bf6bdd333832855692b"
format_stamp: "Formatted at 2026-09-29 20:16:26 on dist-test-slave-4cx3"
server_key: "fd8d4d60aaec48ff0d7c1bf9223079be"
server_key_iv: "517a48e584daf6884fb34cc7136d81ee"
server_key_version: "encryptionkey@0"
I20260929 20:16:26.394203 30360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance:
uuid: "51c98447074e4bf6bdd333832855692b"
format_stamp: "Formatted at 2026-09-29 20:16:26 on dist-test-slave-4cx3"
server_key: "fd8d4d60aaec48ff0d7c1bf9223079be"
server_key_iv: "517a48e584daf6884fb34cc7136d81ee"
server_key_version: "encryptionkey@0"
I20260929 20:16:26.399232 30360 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:16:26.402691 30375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:26.404111 30360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:16:26.404297 30360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "51c98447074e4bf6bdd333832855692b"
format_stamp: "Formatted at 2026-09-29 20:16:26 on dist-test-slave-4cx3"
server_key: "fd8d4d60aaec48ff0d7c1bf9223079be"
server_key_iv: "517a48e584daf6884fb34cc7136d81ee"
server_key_version: "encryptionkey@0"
I20260929 20:16:26.404507 30360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:26.423152 30360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:26.427663 30360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:26.428045 30360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:26.430811 30360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:16:26.430953 30360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:26.431106 30360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:16:26.431214 30360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:16:26 dist-test-slave-4cx3 krb5kdc[30100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712986, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:26.456985 30242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:26.431629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:53153 (local address 127.27.185.126:45305)
0929 20:16:26.431822 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:26.431829 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:26.433302 (+ 1473us) server_negotiation.cc:407] Connection header received
0929 20:16:26.436417 (+ 3115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:26.436424 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:26.436524 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:26.436643 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:26.439112 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:26.440260 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:26.442760 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:26.443195 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:26.443335 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:26.448973 (+ 5638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:26.448981 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:26.448988 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:26.449044 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:26.453100 (+ 4056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:26.454353 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:26.454360 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:26.454365 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:26.454542 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:26.455046 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:26.455052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:26.455055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:26.455335 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:26.455559 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:26.456386 (+ 827us) server_negotiation.cc:299] Negotiation successful
0929 20:16:26.456611 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 20:16:26.458590 30384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:26.432169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:45305 (local address 127.27.185.66:53153)
0929 20:16:26.433115 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:26.433165 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:26.436032 (+ 2867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:26.437104 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:26.437119 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:26.437922 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:26.438884 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:26.438927 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:26.440473 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:26.440482 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:26.442594 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:26.442610 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:26.443461 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:26.443471 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:26.443752 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:26.444653 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:26.444692 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:26.448635 (+ 3943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:26.453305 (+ 4670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:26.453318 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:26.453339 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:26.454189 (+ 850us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:26.454699 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:26.454704 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:26.454707 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:26.454927 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:26.455514 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:26.455523 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:26.455932 (+ 409us) client_negotiation.cc:769] Sending connection context
0929 20:16:26.456265 (+ 333us) client_negotiation.cc:240] Negotiation successful
0929 20:16:26.456871 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":344,"threads_started":1}
I20260929 20:16:26.481412 30476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:26.466781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:45305 (local address 127.27.185.66:47513)
0929 20:16:26.467653 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:26.467681 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:26.467874 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:26.468209 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:26.468214 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:26.468655 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:26.468943 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:26.468952 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:26.470126 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:26.470133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:26.471164 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:26.471175 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:26.472146 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:26.472155 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:26.472516 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:26.473606 (+ 1090us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:26.473638 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:26.474995 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:26.478431 (+ 3436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:26.478438 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:26.478443 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:26.478891 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:26.479343 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:26.479349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:26.479353 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:26.479556 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:26.480212 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:26.480217 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:26.480492 (+ 275us) client_negotiation.cc:769] Sending connection context
0929 20:16:26.480687 (+ 195us) client_negotiation.cc:240] Negotiation successful
0929 20:16:26.481042 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":531,"threads_started":1}
I20260929 20:16:26.481664 30242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:26.466774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:47513 (local address 127.27.185.126:45305)
0929 20:16:26.467053 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:26.467062 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:26.467778 (+ 716us) server_negotiation.cc:407] Connection header received
0929 20:16:26.468002 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:26.468008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:26.468082 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:26.468186 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:26.469098 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:26.469995 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:26.471315 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:26.471954 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:26.472084 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:26.475240 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:26.475247 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:26.475252 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:26.475297 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:26.478279 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:26.479036 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:26.479044 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:26.479048 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:26.479209 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:26.479686 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:26.479694 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:26.479699 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:26.480067 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:26.480241 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:26.480873 (+ 632us) server_negotiation.cc:299] Negotiation successful
0929 20:16:26.481181 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 20:16:26.482853 30360 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:46807
I20260929 20:16:26.482865 30495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:46807 every 8 connection(s)
I20260929 20:16:26.484220 30360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:16:26.486636 30496 heartbeater.cc:347] Connected to a master server at 127.27.185.126:45305
I20260929 20:16:26.486891 30496 heartbeater.cc:464] Registering TS with master...
I20260929 20:16:26.487648 30496 heartbeater.cc:511] Master 127.27.185.126:45305 requested a full tablet report, sending...
I20260929 20:16:26.489084 30145 ts_manager.cc:194] Registered new tserver with Master: 51c98447074e4bf6bdd333832855692b (127.27.185.66:46807)
I20260929 20:16:26.489970 30145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.66@KRBTEST.COM'} at 127.27.185.66:53153
I20260929 20:16:26.493796 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 30360
I20260929 20:16:26.493970 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance
I20260929 20:16:26.494339 28389 external_mini_cluster.cc:1428] Setting key d7a7674a80c662d5275631d3081a5394
WARNING: no policy specified for kudu/127.27.185.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
I20260929 20:16:26.539539 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:0
--local_ip_for_outbound_sockets=127.27.185.67
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:45305
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:40559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:26.877743 30500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:26.878039 30500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:26.878206 30500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:26.885645 30500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:26.885766 30500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:26.885922 30500 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 20:16:26.886003 30500 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:16:26.886127 30500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:16:26.896241 30500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:40559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.185.126:45305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:26.897590 30500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:26.898937 30500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:26.912565 30506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:26.912569 30505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:26.912820 30508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:26.913448 30500 server_base.cc:1034] running on GCE node
I20260929 20:16:26.914067 30500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:26.915071 30500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:26.916349 30500 hybrid_clock.cc:648] HybridClock initialized: now 1790712986916242 us; error 62 us; skew 500 ppm
I20260929 20:16:26.916675 30500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:26 dist-test-slave-4cx3 krb5kdc[30100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712986, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:26.920032 30500 init.cc:377] Logged in from keytab as kudu/127.27.185.67@KRBTEST.COM (short username kudu)
I20260929 20:16:26.921829 30500 webserver.cc:460] Webserver started at http://127.27.185.67:34765/ using document root <none> and password file <none>
I20260929 20:16:26.922654 30500 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:26.922819 30500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:26.923247 30500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:26.926841 30500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-2/data/instance:
uuid: "35549558d566453f8de25d956918d261"
format_stamp: "Formatted at 2026-09-29 20:16:26 on dist-test-slave-4cx3"
server_key: "c4b8e3a001711f3dd4e709f1fcbd09d5"
server_key_iv: "86ced08e392ee34a5359cdbcf562bf2f"
server_key_version: "encryptionkey@0"
I20260929 20:16:26.927757 30500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance:
uuid: "35549558d566453f8de25d956918d261"
format_stamp: "Formatted at 2026-09-29 20:16:26 on dist-test-slave-4cx3"
server_key: "c4b8e3a001711f3dd4e709f1fcbd09d5"
server_key_iv: "86ced08e392ee34a5359cdbcf562bf2f"
server_key_version: "encryptionkey@0"
I20260929 20:16:26.933118 30500 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:16:26.936837 30515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:26.938360 30500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:16:26.938540 30500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "35549558d566453f8de25d956918d261"
format_stamp: "Formatted at 2026-09-29 20:16:26 on dist-test-slave-4cx3"
server_key: "c4b8e3a001711f3dd4e709f1fcbd09d5"
server_key_iv: "86ced08e392ee34a5359cdbcf562bf2f"
server_key_version: "encryptionkey@0"
I20260929 20:16:26.938742 30500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:26.961256 30500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:26.966873 30500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:26.967176 30500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:26.970258 30500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:16:26.970366 30500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:26.970568 30500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:16:26.970685 30500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:26.976536 30356 heartbeater.cc:503] Master 127.27.185.126:45305 was elected leader, sending a full tablet report...
Sep 29 20:16:26 dist-test-slave-4cx3 krb5kdc[30100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712986, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:27.000684 30242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:26.971239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:34245 (local address 127.27.185.126:45305)
0929 20:16:26.971513 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:26.971518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:26.973102 (+ 1584us) server_negotiation.cc:407] Connection header received
0929 20:16:26.976055 (+ 2953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:26.976062 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:26.976175 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:26.976316 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:26.979666 (+ 3350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:26.980816 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:26.984262 (+ 3446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:26.984840 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:26.984986 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:26.990771 (+ 5785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:26.990781 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:26.990789 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:26.990844 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:26.994936 (+ 4092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:26.996293 (+ 1357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:26.996300 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:26.996304 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:26.996483 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:26.997176 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:26.997181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:26.997186 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:26.997591 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:26.997761 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:26.999700 (+ 1939us) server_negotiation.cc:299] Negotiation successful
0929 20:16:27.000064 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 20:16:27.001027 30524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:26.971992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:45305 (local address 127.27.185.67:34245)
0929 20:16:26.972915 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:26.972956 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:26.975658 (+ 2702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:26.977092 (+ 1434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:26.977107 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:26.978174 (+ 1067us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:26.979406 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:26.979447 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:26.981030 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:26.981039 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:26.984052 (+ 3013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:26.984077 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:26.985386 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:26.985397 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:26.985710 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:26.986887 (+ 1177us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:26.986937 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:26.990453 (+ 3516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:26.995170 (+ 4717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:26.995186 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:26.995222 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:26.996092 (+ 870us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:26.996628 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:26.996637 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:26.996639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:26.996993 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:26.997830 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:26.997837 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:26.998490 (+ 653us) client_negotiation.cc:769] Sending connection context
0929 20:16:26.998935 (+ 445us) client_negotiation.cc:240] Negotiation successful
0929 20:16:26.999624 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":369,"threads_started":1}
I20260929 20:16:27.025452 30500 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:40545
I20260929 20:16:27.025470 30634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:40545 every 8 connection(s)
I20260929 20:16:27.026877 30500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:16:27.029282 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 30500
I20260929 20:16:27.029584 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance
I20260929 20:16:27.030141 28389 external_mini_cluster.cc:1428] Setting key ee92c98a2b5b3517fecd23dbd69723ff
I20260929 20:16:27.030721 30635 heartbeater.cc:347] Connected to a master server at 127.27.185.126:45305
I20260929 20:16:27.031118 30635 heartbeater.cc:464] Registering TS with master...
I20260929 20:16:27.031134 30242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:27.013780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:53895 (local address 127.27.185.126:45305)
0929 20:16:27.014058 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:27.014067 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:27.014812 (+ 745us) server_negotiation.cc:407] Connection header received
0929 20:16:27.015066 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:27.015074 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:27.015170 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:27.015284 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:27.017142 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.017911 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.021021 (+ 3110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.021622 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.021732 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:27.024588 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:27.024593 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:27.024597 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:27.024630 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:27.027727 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:27.028713 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:27.028718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:27.028721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:27.028907 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:27.029394 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:27.029399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:27.029403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:27.029810 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:27.029982 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:27.030489 (+ 507us) server_negotiation.cc:299] Negotiation successful
0929 20:16:27.030738 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260929 20:16:27.031206 30613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:27.013805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:45305 (local address 127.27.185.67:53895)
0929 20:16:27.014659 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:27.014690 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:27.014887 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:27.015342 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:27.015347 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:27.016393 (+ 1046us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:27.016939 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:27.016959 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.018073 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.018082 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:27.020768 (+ 2686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:27.020791 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.021846 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.021863 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:27.022038 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:27.022874 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:27.022908 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:27.024339 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:27.027919 (+ 3580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:27.027926 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:27.027932 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:27.028531 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:27.029051 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:27.029056 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:27.029060 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:27.029232 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:27.029982 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:27.029986 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:27.030149 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 20:16:27.030322 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 20:16:27.030666 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":412,"threads_started":1}
I20260929 20:16:27.033273 30635 heartbeater.cc:511] Master 127.27.185.126:45305 requested a full tablet report, sending...
I20260929 20:16:27.034941 30144 ts_manager.cc:194] Registered new tserver with Master: 35549558d566453f8de25d956918d261 (127.27.185.67:40545)
I20260929 20:16:27.035866 30144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.67@KRBTEST.COM'} at 127.27.185.67:34245
I20260929 20:16:27.045958 28389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 20:16:27 dist-test-slave-4cx3 krb5kdc[30100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 20:16:27 dist-test-slave-4cx3 krb5kdc[30100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:27.094017 30242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:27.079954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55036 (local address 127.27.185.126:45305)
0929 20:16:27.080209 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:27.080216 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:27.080403 (+ 187us) server_negotiation.cc:407] Connection header received
0929 20:16:27.080641 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:27.080646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:27.080723 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:27.080836 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:27.082277 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.083047 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.084395 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.084845 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.084983 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:27.089236 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:27.089242 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:27.089245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:27.089292 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:27.091792 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:27.092406 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:27.092410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:27.092412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:27.092534 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:27.092836 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:27.092839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:27.092841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:27.093111 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:27.093287 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:27.093530 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 20:16:27.093707 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 20:16:27.098323 30144 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:16:27.105599 30144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:16:27.129078 30649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:27.117845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:46807 (local address 127.0.0.1:50900)
0929 20:16:27.118844 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:27.118895 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:27.119117 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:27.121176 (+ 2059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:27.121187 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:27.121223 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:16:27.121629 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:27.121643 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.123844 (+ 2201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.123849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:27.126317 (+ 2468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:27.126331 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.128033 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.128040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:27.128232 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:27.128297 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 20:16:27.128448 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 20:16:27.128645 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":459,"threads_started":1}
I20260929 20:16:27.129673 30653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:27.119844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:40545 (local address 127.0.0.1:60838)
0929 20:16:27.120770 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:27.120786 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:27.120889 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:27.122153 (+ 1264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:27.122157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:27.122182 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:16:27.122573 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:27.122585 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.124831 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.124836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:27.126917 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:27.126929 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.128824 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.128832 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:27.128992 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:27.129024 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 20:16:27.129098 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 20:16:27.129244 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"mutex_wait_us":4,"thread_start_us":672,"threads_started":1}
I20260929 20:16:27.130247 30650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:27.118007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:38361 (local address 127.0.0.1:56330)
0929 20:16:27.119806 (+ 1799us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:27.119829 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:27.119957 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:27.122565 (+ 2608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:27.122571 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:27.122599 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:16:27.122955 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:27.122963 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.124595 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.124658 (+ 63us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:27.127086 (+ 2428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:27.127100 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.129401 (+ 2301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.129411 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:27.129603 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:27.129648 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 20:16:27.129752 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 20:16:27.129886 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1646,"mutex_wait_us":93,"thread_start_us":1491,"threads_started":1}
I20260929 20:16:27.130272 30652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:27.118587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56330 (local address 127.27.185.65:38361)
0929 20:16:27.121916 (+ 3329us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:27.121924 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:27.121953 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:16:27.122058 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:27.122065 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:27.122284 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:27.123127 (+ 843us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:16:27.123275 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.124449 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.127256 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.128711 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.128838 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:27.129051 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:27.129796 (+ 745us) server_negotiation.cc:299] Negotiation successful
0929 20:16:27.129953 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3185,"thread_start_us":3054,"threads_started":1}
I20260929 20:16:27.130447 30651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:27.118037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50900 (local address 127.27.185.66:46807)
0929 20:16:27.120478 (+ 2441us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:27.120489 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:27.120520 (+ 31us) server_negotiation.cc:407] Connection header received
0929 20:16:27.120630 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:27.120641 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:27.120925 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:27.121843 (+ 918us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:16:27.122058 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.123616 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.126518 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.127822 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.127986 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:27.128216 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:27.129739 (+ 1523us) server_negotiation.cc:299] Negotiation successful
0929 20:16:27.129934 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2262,"thread_start_us":2136,"threads_started":1}
I20260929 20:16:27.130995 30654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:27.120159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60838 (local address 127.27.185.67:40545)
0929 20:16:27.121546 (+ 1387us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:27.121555 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:27.121590 (+ 35us) server_negotiation.cc:407] Connection header received
0929 20:16:27.121690 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:27.121698 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:27.121942 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:27.122771 (+ 829us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:16:27.123066 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.124673 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.127077 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.128589 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.130114 (+ 1525us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:27.130363 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:27.130460 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 20:16:27.130621 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1202,"thread_start_us":1005,"threads_started":1}
I20260929 20:16:27.134294 30565 tablet_service.cc:1467] Processing CreateTablet for tablet 68df0491b17345088a3bcec6ea8a755b (DEFAULT_TABLE table=test-table [id=afae8b9f4c39466e837563c342c18ce8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:16:27.134323 30425 tablet_service.cc:1467] Processing CreateTablet for tablet 68df0491b17345088a3bcec6ea8a755b (DEFAULT_TABLE table=test-table [id=afae8b9f4c39466e837563c342c18ce8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:16:27.134322 30285 tablet_service.cc:1467] Processing CreateTablet for tablet 68df0491b17345088a3bcec6ea8a755b (DEFAULT_TABLE table=test-table [id=afae8b9f4c39466e837563c342c18ce8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:16:27.135641 30565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68df0491b17345088a3bcec6ea8a755b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:27.135658 30285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68df0491b17345088a3bcec6ea8a755b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:27.135671 30425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68df0491b17345088a3bcec6ea8a755b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:27.147681 30655 tablet_bootstrap.cc:492] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261: Bootstrap starting.
I20260929 20:16:27.150266 30656 tablet_bootstrap.cc:492] T 68df0491b17345088a3bcec6ea8a755b P 51c98447074e4bf6bdd333832855692b: Bootstrap starting.
I20260929 20:16:27.151057 30657 tablet_bootstrap.cc:492] T 68df0491b17345088a3bcec6ea8a755b P 058bb4d2a4d8493c8a15a926674d3a73: Bootstrap starting.
I20260929 20:16:27.152190 30655 tablet_bootstrap.cc:654] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:27.153559 30655 log.cc:824] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261: Log is configured to *not* fsync() on all Append() calls
I20260929 20:16:27.154884 30657 tablet_bootstrap.cc:654] T 68df0491b17345088a3bcec6ea8a755b P 058bb4d2a4d8493c8a15a926674d3a73: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:27.156236 30657 log.cc:824] T 68df0491b17345088a3bcec6ea8a755b P 058bb4d2a4d8493c8a15a926674d3a73: Log is configured to *not* fsync() on all Append() calls
I20260929 20:16:27.156579 30655 tablet_bootstrap.cc:492] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261: No bootstrap required, opened a new log
I20260929 20:16:27.156854 30655 ts_tablet_manager.cc:1397] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 20:16:27.157986 30656 tablet_bootstrap.cc:654] T 68df0491b17345088a3bcec6ea8a755b P 51c98447074e4bf6bdd333832855692b: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:27.159458 30656 log.cc:824] T 68df0491b17345088a3bcec6ea8a755b P 51c98447074e4bf6bdd333832855692b: Log is configured to *not* fsync() on all Append() calls
I20260929 20:16:27.162133 30657 tablet_bootstrap.cc:492] T 68df0491b17345088a3bcec6ea8a755b P 058bb4d2a4d8493c8a15a926674d3a73: No bootstrap required, opened a new log
I20260929 20:16:27.162415 30657 ts_tablet_manager.cc:1397] T 68df0491b17345088a3bcec6ea8a755b P 058bb4d2a4d8493c8a15a926674d3a73: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260929 20:16:27.163786 30656 tablet_bootstrap.cc:492] T 68df0491b17345088a3bcec6ea8a755b P 51c98447074e4bf6bdd333832855692b: No bootstrap required, opened a new log
I20260929 20:16:27.164040 30656 ts_tablet_manager.cc:1397] T 68df0491b17345088a3bcec6ea8a755b P 51c98447074e4bf6bdd333832855692b: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.001s
I20260929 20:16:27.166622 30655 raft_consensus.cc:348] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:27.167091 30655 raft_consensus.cc:374] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:27.167193 30655 raft_consensus.cc:729] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35549558d566453f8de25d956918d261, State: Initialized, Role: FOLLOWER
I20260929 20:16:27.167855 30655 consensus_queue.cc:260] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:27.169028 30635 heartbeater.cc:503] Master 127.27.185.126:45305 was elected leader, sending a full tablet report...
I20260929 20:16:27.169778 30655 ts_tablet_manager.cc:1428] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260929 20:16:27.172729 30657 raft_consensus.cc:348] T 68df0491b17345088a3bcec6ea8a755b P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:27.173120 30657 raft_consensus.cc:374] T 68df0491b17345088a3bcec6ea8a755b P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:27.173245 30657 raft_consensus.cc:729] T 68df0491b17345088a3bcec6ea8a755b P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 058bb4d2a4d8493c8a15a926674d3a73, State: Initialized, Role: FOLLOWER
I20260929 20:16:27.173940 30656 raft_consensus.cc:348] T 68df0491b17345088a3bcec6ea8a755b P 51c98447074e4bf6bdd333832855692b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:27.174049 30657 consensus_queue.cc:260] T 68df0491b17345088a3bcec6ea8a755b P 058bb4d2a4d8493c8a15a926674d3a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:27.174300 30656 raft_consensus.cc:374] T 68df0491b17345088a3bcec6ea8a755b P 51c98447074e4bf6bdd333832855692b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:27.174413 30656 raft_consensus.cc:729] T 68df0491b17345088a3bcec6ea8a755b P 51c98447074e4bf6bdd333832855692b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c98447074e4bf6bdd333832855692b, State: Initialized, Role: FOLLOWER
I20260929 20:16:27.175028 30656 consensus_queue.cc:260] T 68df0491b17345088a3bcec6ea8a755b P 51c98447074e4bf6bdd333832855692b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:27.176241 30657 ts_tablet_manager.cc:1428] T 68df0491b17345088a3bcec6ea8a755b P 058bb4d2a4d8493c8a15a926674d3a73: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260929 20:16:27.182093 30496 heartbeater.cc:503] Master 127.27.185.126:45305 was elected leader, sending a full tablet report...
I20260929 20:16:27.183143 30656 ts_tablet_manager.cc:1428] T 68df0491b17345088a3bcec6ea8a755b P 51c98447074e4bf6bdd333832855692b: Time spent starting tablet: real 0.019s user 0.015s sys 0.003s
I20260929 20:16:27.216281 30661 raft_consensus.cc:482] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:16:27.216746 30661 raft_consensus.cc:504] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
W20260929 20:16:27.216948 30357 tablet.cc:2369] T 68df0491b17345088a3bcec6ea8a755b P 058bb4d2a4d8493c8a15a926674d3a73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:16:27.218552 30661 leader_election.cc:288] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51c98447074e4bf6bdd333832855692b (127.27.185.66:46807), 058bb4d2a4d8493c8a15a926674d3a73 (127.27.185.65:38361)
I20260929 20:16:27.225198 30651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:27.219000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:34783 (local address 127.27.185.66:46807)
0929 20:16:27.219210 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:27.219216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:27.219237 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:16:27.219316 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:27.219319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:27.219452 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:27.219553 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:16:27.220209 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.221164 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.223230 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.224263 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.224444 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:27.224535 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:27.224795 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 20:16:27.224923 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:16:27.225220 30524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:27.218769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:46807 (local address 127.27.185.67:34783)
0929 20:16:27.218976 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:27.218996 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:27.219138 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:27.219746 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:27.219751 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:27.219813 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:16:27.220066 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:27.220078 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.221319 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.221324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:27.223073 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:27.223086 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.224424 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.224430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:27.224631 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:27.224656 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 20:16:27.224769 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 20:16:27.224910 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":1}
I20260929 20:16:27.225612 30652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:27.219475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:39215 (local address 127.27.185.65:38361)
0929 20:16:27.219670 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:27.219676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:27.219955 (+ 279us) server_negotiation.cc:407] Connection header received
0929 20:16:27.220122 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:27.220126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:27.220221 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:27.220321 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:16:27.220867 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.221829 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.223835 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.224698 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.224877 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:27.224957 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:27.225241 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 20:16:27.225368 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:16:27.225728 30664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:27.219040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:38361 (local address 127.27.185.67:39215)
0929 20:16:27.219851 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:27.219873 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:27.219989 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:27.220344 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:27.220348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:27.220378 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:16:27.220711 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:27.220724 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.221976 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.221985 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:27.223683 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:27.223695 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.224933 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.224939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:27.225093 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:27.225115 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 20:16:27.225222 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 20:16:27.225342 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":36,"thread_start_us":476,"threads_started":1}
I20260929 20:16:27.226492 30445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68df0491b17345088a3bcec6ea8a755b" candidate_uuid: "35549558d566453f8de25d956918d261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51c98447074e4bf6bdd333832855692b" is_pre_election: true
I20260929 20:16:27.226683 30305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68df0491b17345088a3bcec6ea8a755b" candidate_uuid: "35549558d566453f8de25d956918d261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "058bb4d2a4d8493c8a15a926674d3a73" is_pre_election: true
I20260929 20:16:27.227126 30445 raft_consensus.cc:2454] T 68df0491b17345088a3bcec6ea8a755b P 51c98447074e4bf6bdd333832855692b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35549558d566453f8de25d956918d261 in term 0.
I20260929 20:16:27.227159 30305 raft_consensus.cc:2454] T 68df0491b17345088a3bcec6ea8a755b P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35549558d566453f8de25d956918d261 in term 0.
I20260929 20:16:27.227844 30519 leader_election.cc:302] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 058bb4d2a4d8493c8a15a926674d3a73, 35549558d566453f8de25d956918d261; no voters:
I20260929 20:16:27.228258 30661 raft_consensus.cc:2781] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:16:27.228417 30661 raft_consensus.cc:482] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:16:27.228556 30661 raft_consensus.cc:3037] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:27.230769 30661 raft_consensus.cc:504] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:27.231426 30661 leader_election.cc:288] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261 [CANDIDATE]: Term 1 election: Requested vote from peers 51c98447074e4bf6bdd333832855692b (127.27.185.66:46807), 058bb4d2a4d8493c8a15a926674d3a73 (127.27.185.65:38361)
I20260929 20:16:27.231839 30305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68df0491b17345088a3bcec6ea8a755b" candidate_uuid: "35549558d566453f8de25d956918d261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "058bb4d2a4d8493c8a15a926674d3a73"
I20260929 20:16:27.231865 30445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68df0491b17345088a3bcec6ea8a755b" candidate_uuid: "35549558d566453f8de25d956918d261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51c98447074e4bf6bdd333832855692b"
I20260929 20:16:27.232048 30305 raft_consensus.cc:3037] T 68df0491b17345088a3bcec6ea8a755b P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:27.232050 30445 raft_consensus.cc:3037] T 68df0491b17345088a3bcec6ea8a755b P 51c98447074e4bf6bdd333832855692b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:27.234447 30445 raft_consensus.cc:2454] T 68df0491b17345088a3bcec6ea8a755b P 51c98447074e4bf6bdd333832855692b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35549558d566453f8de25d956918d261 in term 1.
I20260929 20:16:27.234447 30305 raft_consensus.cc:2454] T 68df0491b17345088a3bcec6ea8a755b P 058bb4d2a4d8493c8a15a926674d3a73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35549558d566453f8de25d956918d261 in term 1.
I20260929 20:16:27.234902 30518 leader_election.cc:302] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35549558d566453f8de25d956918d261, 51c98447074e4bf6bdd333832855692b; no voters:
I20260929 20:16:27.235208 30661 raft_consensus.cc:2781] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:16:27.235707 30661 raft_consensus.cc:686] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261 [term 1 LEADER]: Becoming Leader. State: Replica: 35549558d566453f8de25d956918d261, State: Running, Role: LEADER
I20260929 20:16:27.236191 30661 consensus_queue.cc:237] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
W20260929 20:16:27.236559 30497 tablet.cc:2369] T 68df0491b17345088a3bcec6ea8a755b P 51c98447074e4bf6bdd333832855692b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:16:27.240736 30144 catalog_manager.cc:5495] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35549558d566453f8de25d956918d261 (127.27.185.67). New cstate: current_term: 1 leader_uuid: "35549558d566453f8de25d956918d261" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } health_report { overall_health: HEALTHY } } }
W20260929 20:16:27.278904 30637 tablet.cc:2369] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 20:16:27 dist-test-slave-4cx3 krb5kdc[30100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712985, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:27.294674 30672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:27.278451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:45305 (local address 127.0.0.1:55050)
0929 20:16:27.279226 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:27.279249 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:27.279481 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:27.279940 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:27.279943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:27.280675 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:27.280978 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:27.280988 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.282429 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.282433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:27.283595 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:27.283608 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.284611 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.284616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:27.284760 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:27.285527 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:27.285545 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:27.288251 (+ 2706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:27.292033 (+ 3782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:27.292044 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:27.292050 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:27.292533 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:27.292948 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:27.292951 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:27.292953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:27.293088 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:27.293777 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:27.293784 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:27.293952 (+ 168us) client_negotiation.cc:769] Sending connection context
0929 20:16:27.294104 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 20:16:27.294323 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":348,"threads_started":1}
I20260929 20:16:27.295154 30242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:27.278336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55050 (local address 127.27.185.126:45305)
0929 20:16:27.278576 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:27.278583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:27.279369 (+ 786us) server_negotiation.cc:407] Connection header received
0929 20:16:27.279646 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:27.279656 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:27.279751 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:27.279877 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:27.281122 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.282290 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.283739 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.284422 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.284544 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:27.288459 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:27.288466 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:27.288471 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:27.288508 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:27.291882 (+ 3374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:27.292663 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:27.292670 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:27.292677 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:27.292845 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:27.293189 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:27.293194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:27.293198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:27.293573 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:27.293753 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:27.294276 (+ 523us) server_negotiation.cc:299] Negotiation successful
0929 20:16:27.294501 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:16:27.300622 30144 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:55050:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 20:16:27.301676 30144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:16:27.308597 30285 tablet_service.cc:1467] Processing CreateTablet for tablet 2bef558f5ac24ebda04e4e5a9e853385 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c4ff687ac35a40a49818a2d5ee2d90f5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:16:27.309048 30425 tablet_service.cc:1467] Processing CreateTablet for tablet 2bef558f5ac24ebda04e4e5a9e853385 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c4ff687ac35a40a49818a2d5ee2d90f5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:16:27.309060 30565 tablet_service.cc:1467] Processing CreateTablet for tablet 2bef558f5ac24ebda04e4e5a9e853385 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c4ff687ac35a40a49818a2d5ee2d90f5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:16:27.309725 30285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bef558f5ac24ebda04e4e5a9e853385. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:27.309904 30425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bef558f5ac24ebda04e4e5a9e853385. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:27.309930 30565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bef558f5ac24ebda04e4e5a9e853385. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:27.315030 30656 tablet_bootstrap.cc:492] T 2bef558f5ac24ebda04e4e5a9e853385 P 51c98447074e4bf6bdd333832855692b: Bootstrap starting.
I20260929 20:16:27.316453 30656 tablet_bootstrap.cc:654] T 2bef558f5ac24ebda04e4e5a9e853385 P 51c98447074e4bf6bdd333832855692b: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:27.317067 30657 tablet_bootstrap.cc:492] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73: Bootstrap starting.
I20260929 20:16:27.317560 30655 tablet_bootstrap.cc:492] T 2bef558f5ac24ebda04e4e5a9e853385 P 35549558d566453f8de25d956918d261: Bootstrap starting.
I20260929 20:16:27.318567 30657 tablet_bootstrap.cc:654] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:27.319651 30655 tablet_bootstrap.cc:654] T 2bef558f5ac24ebda04e4e5a9e853385 P 35549558d566453f8de25d956918d261: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:27.322808 30655 tablet_bootstrap.cc:492] T 2bef558f5ac24ebda04e4e5a9e853385 P 35549558d566453f8de25d956918d261: No bootstrap required, opened a new log
I20260929 20:16:27.322969 30655 ts_tablet_manager.cc:1397] T 2bef558f5ac24ebda04e4e5a9e853385 P 35549558d566453f8de25d956918d261: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260929 20:16:27.323259 30657 tablet_bootstrap.cc:492] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73: No bootstrap required, opened a new log
I20260929 20:16:27.323431 30657 ts_tablet_manager.cc:1397] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s
I20260929 20:16:27.323913 30655 raft_consensus.cc:348] T 2bef558f5ac24ebda04e4e5a9e853385 P 35549558d566453f8de25d956918d261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:27.324182 30655 raft_consensus.cc:374] T 2bef558f5ac24ebda04e4e5a9e853385 P 35549558d566453f8de25d956918d261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:27.324283 30655 raft_consensus.cc:729] T 2bef558f5ac24ebda04e4e5a9e853385 P 35549558d566453f8de25d956918d261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35549558d566453f8de25d956918d261, State: Initialized, Role: FOLLOWER
I20260929 20:16:27.324265 30657 raft_consensus.cc:348] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:27.324564 30657 raft_consensus.cc:374] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:27.324656 30657 raft_consensus.cc:729] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 058bb4d2a4d8493c8a15a926674d3a73, State: Initialized, Role: FOLLOWER
I20260929 20:16:27.324520 30655 consensus_queue.cc:260] T 2bef558f5ac24ebda04e4e5a9e853385 P 35549558d566453f8de25d956918d261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:27.324901 30657 consensus_queue.cc:260] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:27.325179 30655 ts_tablet_manager.cc:1428] T 2bef558f5ac24ebda04e4e5a9e853385 P 35549558d566453f8de25d956918d261: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260929 20:16:27.325379 30662 tablet_replica.cc:439] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } } }
I20260929 20:16:27.325819 30662 tablet_replica.cc:442] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:16:27.326148 30661 tablet_replica.cc:439] T 2bef558f5ac24ebda04e4e5a9e853385 P 35549558d566453f8de25d956918d261: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } } }
I20260929 20:16:27.326439 30661 tablet_replica.cc:442] T 2bef558f5ac24ebda04e4e5a9e853385 P 35549558d566453f8de25d956918d261: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:16:27.326457 30657 ts_tablet_manager.cc:1428] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260929 20:16:27.329286 30656 tablet_bootstrap.cc:492] T 2bef558f5ac24ebda04e4e5a9e853385 P 51c98447074e4bf6bdd333832855692b: No bootstrap required, opened a new log
I20260929 20:16:27.329455 30656 ts_tablet_manager.cc:1397] T 2bef558f5ac24ebda04e4e5a9e853385 P 51c98447074e4bf6bdd333832855692b: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.006s
I20260929 20:16:27.330149 30656 raft_consensus.cc:348] T 2bef558f5ac24ebda04e4e5a9e853385 P 51c98447074e4bf6bdd333832855692b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:27.330404 30656 raft_consensus.cc:374] T 2bef558f5ac24ebda04e4e5a9e853385 P 51c98447074e4bf6bdd333832855692b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:27.330483 30656 raft_consensus.cc:729] T 2bef558f5ac24ebda04e4e5a9e853385 P 51c98447074e4bf6bdd333832855692b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c98447074e4bf6bdd333832855692b, State: Initialized, Role: FOLLOWER
I20260929 20:16:27.330722 30656 consensus_queue.cc:260] T 2bef558f5ac24ebda04e4e5a9e853385 P 51c98447074e4bf6bdd333832855692b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:27.331324 30656 ts_tablet_manager.cc:1428] T 2bef558f5ac24ebda04e4e5a9e853385 P 51c98447074e4bf6bdd333832855692b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 20:16:27.331153 30663 tablet_replica.cc:439] T 2bef558f5ac24ebda04e4e5a9e853385 P 51c98447074e4bf6bdd333832855692b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } } }
I20260929 20:16:27.331584 30663 tablet_replica.cc:442] T 2bef558f5ac24ebda04e4e5a9e853385 P 51c98447074e4bf6bdd333832855692b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:16:27.586256 30662 raft_consensus.cc:482] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:16:27.586537 30662 raft_consensus.cc:504] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:27.588321 30662 leader_election.cc:288] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51c98447074e4bf6bdd333832855692b (127.27.185.66:46807), 35549558d566453f8de25d956918d261 (127.27.185.67:40545)
I20260929 20:16:27.595008 30651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:27.588667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:60317 (local address 127.27.185.66:46807)
0929 20:16:27.589028 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:27.589037 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:27.589391 (+ 354us) server_negotiation.cc:407] Connection header received
0929 20:16:27.589598 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:27.589606 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:27.589701 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:27.589794 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:16:27.590470 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.591374 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.593225 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.594012 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.594199 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:27.594277 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:27.594569 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 20:16:27.594695 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205}
I20260929 20:16:27.595188 30676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:27.588532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:46807 (local address 127.27.185.65:60317)
0929 20:16:27.589223 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:27.589246 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:27.589447 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:27.589829 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:27.589834 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:27.589860 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:16:27.590299 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:27.590310 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.591533 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.591537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:27.593052 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:27.593065 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.594221 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.594227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:27.594396 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:27.594427 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 20:16:27.594534 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 20:16:27.594670 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":28,"thread_start_us":352,"threads_started":1}
I20260929 20:16:27.595232 30654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:27.588875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:39341 (local address 127.27.185.67:40545)
0929 20:16:27.589060 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:27.589067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:27.589658 (+ 591us) server_negotiation.cc:407] Connection header received
0929 20:16:27.589833 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:27.589838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:27.589915 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:27.590015 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:16:27.590582 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.591254 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.593182 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.594006 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.594218 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:27.594324 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:27.594734 (+ 410us) server_negotiation.cc:299] Negotiation successful
0929 20:16:27.594868 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:16:27.595274 30677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:27.588667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:40545 (local address 127.27.185.65:39341)
0929 20:16:27.589535 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:27.589559 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:27.589685 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:27.590040 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:27.590043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:27.590065 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:16:27.590375 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:27.590413 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.591435 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.591444 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:27.593020 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:27.593032 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.594451 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.594456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:27.594573 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:27.594596 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:16:27.594706 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 20:16:27.594837 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":2,"thread_start_us":522,"threads_started":1}
I20260929 20:16:27.596210 30445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bef558f5ac24ebda04e4e5a9e853385" candidate_uuid: "058bb4d2a4d8493c8a15a926674d3a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51c98447074e4bf6bdd333832855692b" is_pre_election: true
I20260929 20:16:27.596202 30585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bef558f5ac24ebda04e4e5a9e853385" candidate_uuid: "058bb4d2a4d8493c8a15a926674d3a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35549558d566453f8de25d956918d261" is_pre_election: true
I20260929 20:16:27.596499 30445 raft_consensus.cc:2454] T 2bef558f5ac24ebda04e4e5a9e853385 P 51c98447074e4bf6bdd333832855692b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 058bb4d2a4d8493c8a15a926674d3a73 in term 0.
I20260929 20:16:27.596554 30585 raft_consensus.cc:2454] T 2bef558f5ac24ebda04e4e5a9e853385 P 35549558d566453f8de25d956918d261 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 058bb4d2a4d8493c8a15a926674d3a73 in term 0.
I20260929 20:16:27.597023 30236 leader_election.cc:302] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 058bb4d2a4d8493c8a15a926674d3a73, 51c98447074e4bf6bdd333832855692b; no voters:
I20260929 20:16:27.597421 30662 raft_consensus.cc:2781] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:16:27.597550 30662 raft_consensus.cc:482] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:16:27.597652 30662 raft_consensus.cc:3037] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:27.599799 30662 raft_consensus.cc:504] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:27.600427 30662 leader_election.cc:288] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73 [CANDIDATE]: Term 1 election: Requested vote from peers 51c98447074e4bf6bdd333832855692b (127.27.185.66:46807), 35549558d566453f8de25d956918d261 (127.27.185.67:40545)
I20260929 20:16:27.600822 30445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bef558f5ac24ebda04e4e5a9e853385" candidate_uuid: "058bb4d2a4d8493c8a15a926674d3a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51c98447074e4bf6bdd333832855692b"
I20260929 20:16:27.600926 30585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bef558f5ac24ebda04e4e5a9e853385" candidate_uuid: "058bb4d2a4d8493c8a15a926674d3a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35549558d566453f8de25d956918d261"
I20260929 20:16:27.601040 30445 raft_consensus.cc:3037] T 2bef558f5ac24ebda04e4e5a9e853385 P 51c98447074e4bf6bdd333832855692b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:27.601166 30585 raft_consensus.cc:3037] T 2bef558f5ac24ebda04e4e5a9e853385 P 35549558d566453f8de25d956918d261 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:27.603322 30445 raft_consensus.cc:2454] T 2bef558f5ac24ebda04e4e5a9e853385 P 51c98447074e4bf6bdd333832855692b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 058bb4d2a4d8493c8a15a926674d3a73 in term 1.
I20260929 20:16:27.603358 30585 raft_consensus.cc:2454] T 2bef558f5ac24ebda04e4e5a9e853385 P 35549558d566453f8de25d956918d261 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 058bb4d2a4d8493c8a15a926674d3a73 in term 1.
I20260929 20:16:27.603788 30236 leader_election.cc:302] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 058bb4d2a4d8493c8a15a926674d3a73, 51c98447074e4bf6bdd333832855692b; no voters:
I20260929 20:16:27.604091 30662 raft_consensus.cc:2781] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:16:27.604537 30662 raft_consensus.cc:686] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73 [term 1 LEADER]: Becoming Leader. State: Replica: 058bb4d2a4d8493c8a15a926674d3a73, State: Running, Role: LEADER
I20260929 20:16:27.605024 30662 consensus_queue.cc:237] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:27.608268 30678 tablet_replica.cc:439] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73: TxnStatusTablet state changed. Reason: New leader 058bb4d2a4d8493c8a15a926674d3a73. Latest consensus state: current_term: 1 leader_uuid: "058bb4d2a4d8493c8a15a926674d3a73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } } }
I20260929 20:16:27.608686 30678 tablet_replica.cc:442] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:16:27.609386 30680 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 20:16:27.609686 30680 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 20:16:27.609961 30145 catalog_manager.cc:5495] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 058bb4d2a4d8493c8a15a926674d3a73 (127.27.185.65). New cstate: current_term: 1 leader_uuid: "058bb4d2a4d8493c8a15a926674d3a73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } health_report { overall_health: UNKNOWN } } }
I20260929 20:16:27.674403 30661 consensus_queue.cc:1035] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:16:27.697209 30661 consensus_queue.cc:1035] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261 [LEADER]: Connected to new peer: Peer: permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:16:27.701167 30672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:27.693103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:38361 (local address 127.0.0.1:56340)
0929 20:16:27.693554 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:27.693576 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:27.693715 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:27.694826 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:27.694831 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:27.694843 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:16:27.695063 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:27.695080 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.696739 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.696744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:27.698218 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:27.698233 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.699282 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.699288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:27.699508 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:27.699594 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:16:27.700606 (+ 1012us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:16:27.700629 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:16:27.700717 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 20:16:27.700875 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260}
I20260929 20:16:27.701191 30652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:27.693964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56340 (local address 127.27.185.65:38361)
0929 20:16:27.694188 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:27.694195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:27.694215 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:16:27.694335 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:27.694339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:27.694454 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:27.694612 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:16:27.695244 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.696494 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.698391 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.698992 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.699123 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:27.699838 (+ 715us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:16:27.700483 (+ 645us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:16:27.700654 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:27.700735 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 20:16:27.700832 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 20:16:27.711342 30445 raft_consensus.cc:1261] T 2bef558f5ac24ebda04e4e5a9e853385 P 51c98447074e4bf6bdd333832855692b [term 1 FOLLOWER]: Refusing update from remote peer 058bb4d2a4d8493c8a15a926674d3a73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:16:27.711351 30585 raft_consensus.cc:1261] T 2bef558f5ac24ebda04e4e5a9e853385 P 35549558d566453f8de25d956918d261 [term 1 FOLLOWER]: Refusing update from remote peer 058bb4d2a4d8493c8a15a926674d3a73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:16:27.712271 30678 consensus_queue.cc:1035] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:16:27.712646 30662 consensus_queue.cc:1035] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:16:27.718883 30661 tablet_replica.cc:439] T 2bef558f5ac24ebda04e4e5a9e853385 P 35549558d566453f8de25d956918d261: TxnStatusTablet state changed. Reason: New leader 058bb4d2a4d8493c8a15a926674d3a73. Latest consensus state: current_term: 1 leader_uuid: "058bb4d2a4d8493c8a15a926674d3a73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } } }
I20260929 20:16:27.719215 30661 tablet_replica.cc:442] T 2bef558f5ac24ebda04e4e5a9e853385 P 35549558d566453f8de25d956918d261: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:16:27.720891 30687 mvcc.cc:205] Tried to move back new op lower bound from 7334760397652799488 to 7334760397233590272. Current Snapshot: MvccSnapshot[applied={T|T < 7334760397652799488}]
I20260929 20:16:27.721549 30683 mvcc.cc:205] Tried to move back new op lower bound from 7334760397652799488 to 7334760397233590272. Current Snapshot: MvccSnapshot[applied={T|T < 7334760397652799488}]
I20260929 20:16:27.725936 30661 tablet_replica.cc:439] T 2bef558f5ac24ebda04e4e5a9e853385 P 35549558d566453f8de25d956918d261: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "058bb4d2a4d8493c8a15a926674d3a73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } } }
I20260929 20:16:27.726225 30661 tablet_replica.cc:442] T 2bef558f5ac24ebda04e4e5a9e853385 P 35549558d566453f8de25d956918d261: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:16:27.729477 30662 tablet_replica.cc:439] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "058bb4d2a4d8493c8a15a926674d3a73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } } }
I20260929 20:16:27.729776 30662 tablet_replica.cc:442] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:16:27.731216 30663 tablet_replica.cc:439] T 2bef558f5ac24ebda04e4e5a9e853385 P 51c98447074e4bf6bdd333832855692b: TxnStatusTablet state changed. Reason: New leader 058bb4d2a4d8493c8a15a926674d3a73. Latest consensus state: current_term: 1 leader_uuid: "058bb4d2a4d8493c8a15a926674d3a73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } } }
I20260929 20:16:27.731552 30663 tablet_replica.cc:442] T 2bef558f5ac24ebda04e4e5a9e853385 P 51c98447074e4bf6bdd333832855692b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:16:27.734099 30678 tablet_replica.cc:439] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "058bb4d2a4d8493c8a15a926674d3a73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } } }
I20260929 20:16:27.734458 30678 tablet_replica.cc:442] T 2bef558f5ac24ebda04e4e5a9e853385 P 058bb4d2a4d8493c8a15a926674d3a73: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:16:27.737591 30663 tablet_replica.cc:439] T 2bef558f5ac24ebda04e4e5a9e853385 P 51c98447074e4bf6bdd333832855692b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "058bb4d2a4d8493c8a15a926674d3a73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } } }
I20260929 20:16:27.737843 30663 tablet_replica.cc:442] T 2bef558f5ac24ebda04e4e5a9e853385 P 51c98447074e4bf6bdd333832855692b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:16:27.749950 30654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:27.741288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60840 (local address 127.27.185.67:40545)
0929 20:16:27.741514 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:27.741524 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:27.744018 (+ 2494us) server_negotiation.cc:407] Connection header received
0929 20:16:27.744221 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:27.744228 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:27.744347 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:27.744521 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:16:27.744881 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.745979 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.747372 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.748022 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.748157 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:27.748540 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:16:27.749284 (+ 744us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:16:27.749434 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:27.749539 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 20:16:27.749633 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:16:27.766793 30652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:27.761256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:45513 (local address 127.27.185.65:38361)
0929 20:16:27.761463 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:27.761472 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:27.761963 (+ 491us) server_negotiation.cc:407] Connection header received
0929 20:16:27.762178 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:27.762182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:27.762251 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:27.762355 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:16:27.762823 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.763642 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.764990 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.765461 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.765600 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:27.766034 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:16:27.766138 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:16:27.766237 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:27.766376 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 20:16:27.766461 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 20:16:27.766857 30699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:27.761144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:38361 (local address 127.27.185.67:45513)
0929 20:16:27.761851 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:27.761866 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:27.762034 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:27.762374 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:27.762378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:27.762390 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:16:27.762659 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:27.762668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.763765 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.763772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:27.764845 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:27.764855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.765623 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.765629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:27.765767 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:27.765822 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:16:27.766238 (+ 416us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:16:27.766251 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 20:16:27.766350 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 20:16:27.766492 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":434,"threads_started":1}
I20260929 20:16:27.808130 30654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:27.800223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:47639 (local address 127.27.185.67:40545)
0929 20:16:27.800390 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:27.800394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:27.801159 (+ 765us) server_negotiation.cc:407] Connection header received
0929 20:16:27.801443 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:27.801450 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:27.801541 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:27.801636 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:16:27.802365 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.803424 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.805834 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.806518 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.806644 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:27.807309 (+ 665us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:16:27.807497 (+ 188us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:16:27.807580 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:27.807745 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 20:16:27.807809 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 20:16:27.808391 30702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:27.800339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:40545 (local address 127.27.185.65:47639)
0929 20:16:27.801058 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:27.801079 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:27.801283 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:27.801684 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:27.801690 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:27.801706 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:16:27.802199 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:27.802213 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.803587 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.803594 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:27.805658 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:27.805677 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:27.806737 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:27.806745 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:27.807020 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:27.807108 (+ 88us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:16:27.807611 (+ 503us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:16:27.807625 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 20:16:27.807734 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 20:16:27.807903 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":423,"threads_started":1}
I20260929 20:16:27.844095 30144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:16:27.844424 30144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:16:27.848639 30144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34f1fba2fa284ab08e77a76867895f77: Sending DeleteTablet for 3 replicas of tablet 68df0491b17345088a3bcec6ea8a755b
I20260929 20:16:27.849840 30425 tablet_service.cc:1514] Processing DeleteTablet for tablet 68df0491b17345088a3bcec6ea8a755b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:16:27 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:50900
I20260929 20:16:27.849884 30285 tablet_service.cc:1514] Processing DeleteTablet for tablet 68df0491b17345088a3bcec6ea8a755b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:16:27 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:56330
I20260929 20:16:27.851055 30565 tablet_service.cc:1514] Processing DeleteTablet for tablet 68df0491b17345088a3bcec6ea8a755b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:16:27 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:60838
I20260929 20:16:27.856410 30145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:16:27.857125 30145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:16:27.863538 30706 tablet_replica.cc:330] T 68df0491b17345088a3bcec6ea8a755b P 51c98447074e4bf6bdd333832855692b: stopping tablet replica
I20260929 20:16:27.866554 30564 tablet_service.cc:1467] Processing CreateTablet for tablet 12531195252640d18c3738a2c8053e61 (DEFAULT_TABLE table=test-table [id=cb8be02b0f6c483096114695ebd778d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:16:27.867147 30564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12531195252640d18c3738a2c8053e61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:27.867156 30425 tablet_service.cc:1467] Processing CreateTablet for tablet 12531195252640d18c3738a2c8053e61 (DEFAULT_TABLE table=test-table [id=cb8be02b0f6c483096114695ebd778d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:16:27.867838 30425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12531195252640d18c3738a2c8053e61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:27.870196 30284 tablet_service.cc:1467] Processing CreateTablet for tablet 12531195252640d18c3738a2c8053e61 (DEFAULT_TABLE table=test-table [id=cb8be02b0f6c483096114695ebd778d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:16:27.870671 30284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12531195252640d18c3738a2c8053e61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:27.875349 30706 raft_consensus.cc:2229] T 68df0491b17345088a3bcec6ea8a755b P 51c98447074e4bf6bdd333832855692b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:16:27.876238 30706 raft_consensus.cc:2258] T 68df0491b17345088a3bcec6ea8a755b P 51c98447074e4bf6bdd333832855692b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:16:27.876555 30707 tablet_replica.cc:330] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261: stopping tablet replica
I20260929 20:16:27.878770 30706 ts_tablet_manager.cc:1905] T 68df0491b17345088a3bcec6ea8a755b P 51c98447074e4bf6bdd333832855692b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:16:27.877350 30707 raft_consensus.cc:2229] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:16:27.883476 30707 raft_consensus.cc:2258] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:16:27.884183 30705 tablet_replica.cc:330] T 68df0491b17345088a3bcec6ea8a755b P 058bb4d2a4d8493c8a15a926674d3a73: stopping tablet replica
I20260929 20:16:27.885004 30705 raft_consensus.cc:2229] T 68df0491b17345088a3bcec6ea8a755b P 058bb4d2a4d8493c8a15a926674d3a73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:16:27.885453 30705 raft_consensus.cc:2258] T 68df0491b17345088a3bcec6ea8a755b P 058bb4d2a4d8493c8a15a926674d3a73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:16:27.886294 30707 ts_tablet_manager.cc:1905] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:16:27.889436 30705 ts_tablet_manager.cc:1905] T 68df0491b17345088a3bcec6ea8a755b P 058bb4d2a4d8493c8a15a926674d3a73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:16:27.894129 30707 ts_tablet_manager.cc:1918] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:16:27.894381 30707 log.cc:1196] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-2/wal/wals/68df0491b17345088a3bcec6ea8a755b
I20260929 20:16:27.895040 30707 ts_tablet_manager.cc:1939] T 68df0491b17345088a3bcec6ea8a755b P 35549558d566453f8de25d956918d261: Deleting consensus metadata
I20260929 20:16:27.897130 30132 catalog_manager.cc:4841] TS 35549558d566453f8de25d956918d261 (127.27.185.67:40545): tablet 68df0491b17345088a3bcec6ea8a755b (table test-table [id=afae8b9f4c39466e837563c342c18ce8]) successfully deleted
I20260929 20:16:27.899192 30706 ts_tablet_manager.cc:1918] T 68df0491b17345088a3bcec6ea8a755b P 51c98447074e4bf6bdd333832855692b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:16:27.899303 30709 tablet_bootstrap.cc:492] T 12531195252640d18c3738a2c8053e61 P 35549558d566453f8de25d956918d261: Bootstrap starting.
I20260929 20:16:27.899446 30706 log.cc:1196] T 68df0491b17345088a3bcec6ea8a755b P 51c98447074e4bf6bdd333832855692b: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-1/wal/wals/68df0491b17345088a3bcec6ea8a755b
I20260929 20:16:27.900064 30706 ts_tablet_manager.cc:1939] T 68df0491b17345088a3bcec6ea8a755b P 51c98447074e4bf6bdd333832855692b: Deleting consensus metadata
I20260929 20:16:27.901500 30709 tablet_bootstrap.cc:654] T 12531195252640d18c3738a2c8053e61 P 35549558d566453f8de25d956918d261: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:27.902058 30131 catalog_manager.cc:4841] TS 51c98447074e4bf6bdd333832855692b (127.27.185.66:46807): tablet 68df0491b17345088a3bcec6ea8a755b (table test-table [id=afae8b9f4c39466e837563c342c18ce8]) successfully deleted
I20260929 20:16:27.904284 30708 tablet_bootstrap.cc:492] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73: Bootstrap starting.
I20260929 20:16:27.904997 30709 tablet_bootstrap.cc:492] T 12531195252640d18c3738a2c8053e61 P 35549558d566453f8de25d956918d261: No bootstrap required, opened a new log
I20260929 20:16:27.905153 30709 ts_tablet_manager.cc:1397] T 12531195252640d18c3738a2c8053e61 P 35549558d566453f8de25d956918d261: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 20:16:27.905961 30709 raft_consensus.cc:348] T 12531195252640d18c3738a2c8053e61 P 35549558d566453f8de25d956918d261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:27.906212 30709 raft_consensus.cc:374] T 12531195252640d18c3738a2c8053e61 P 35549558d566453f8de25d956918d261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:27.906303 30709 raft_consensus.cc:729] T 12531195252640d18c3738a2c8053e61 P 35549558d566453f8de25d956918d261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35549558d566453f8de25d956918d261, State: Initialized, Role: FOLLOWER
I20260929 20:16:27.906672 30708 tablet_bootstrap.cc:654] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:27.906524 30709 consensus_queue.cc:260] T 12531195252640d18c3738a2c8053e61 P 35549558d566453f8de25d956918d261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:27.907167 30709 ts_tablet_manager.cc:1428] T 12531195252640d18c3738a2c8053e61 P 35549558d566453f8de25d956918d261: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 20:16:27.915365 30705 ts_tablet_manager.cc:1918] T 68df0491b17345088a3bcec6ea8a755b P 058bb4d2a4d8493c8a15a926674d3a73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:16:27.917502 30705 log.cc:1196] T 68df0491b17345088a3bcec6ea8a755b P 058bb4d2a4d8493c8a15a926674d3a73: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-0/wal/wals/68df0491b17345088a3bcec6ea8a755b
I20260929 20:16:27.918323 30705 ts_tablet_manager.cc:1939] T 68df0491b17345088a3bcec6ea8a755b P 058bb4d2a4d8493c8a15a926674d3a73: Deleting consensus metadata
I20260929 20:16:27.924606 30132 catalog_manager.cc:4841] TS 058bb4d2a4d8493c8a15a926674d3a73 (127.27.185.65:38361): tablet 68df0491b17345088a3bcec6ea8a755b (table test-table [id=afae8b9f4c39466e837563c342c18ce8]) successfully deleted
I20260929 20:16:27.925168 30710 tablet_bootstrap.cc:492] T 12531195252640d18c3738a2c8053e61 P 51c98447074e4bf6bdd333832855692b: Bootstrap starting.
I20260929 20:16:27.927265 30710 tablet_bootstrap.cc:654] T 12531195252640d18c3738a2c8053e61 P 51c98447074e4bf6bdd333832855692b: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:27.929927 30708 tablet_bootstrap.cc:492] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73: No bootstrap required, opened a new log
I20260929 20:16:27.930096 30708 ts_tablet_manager.cc:1397] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.018s
I20260929 20:16:27.931003 30708 raft_consensus.cc:348] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:27.931262 30708 raft_consensus.cc:374] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:27.931367 30708 raft_consensus.cc:729] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 058bb4d2a4d8493c8a15a926674d3a73, State: Initialized, Role: FOLLOWER
I20260929 20:16:27.931671 30708 consensus_queue.cc:260] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:27.933583 30710 tablet_bootstrap.cc:492] T 12531195252640d18c3738a2c8053e61 P 51c98447074e4bf6bdd333832855692b: No bootstrap required, opened a new log
I20260929 20:16:27.933826 30710 ts_tablet_manager.cc:1397] T 12531195252640d18c3738a2c8053e61 P 51c98447074e4bf6bdd333832855692b: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260929 20:16:27.934756 30710 raft_consensus.cc:348] T 12531195252640d18c3738a2c8053e61 P 51c98447074e4bf6bdd333832855692b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:27.934906 30708 ts_tablet_manager.cc:1428] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260929 20:16:27.935061 30710 raft_consensus.cc:374] T 12531195252640d18c3738a2c8053e61 P 51c98447074e4bf6bdd333832855692b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:27.935205 30710 raft_consensus.cc:729] T 12531195252640d18c3738a2c8053e61 P 51c98447074e4bf6bdd333832855692b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c98447074e4bf6bdd333832855692b, State: Initialized, Role: FOLLOWER
I20260929 20:16:27.936944 30710 consensus_queue.cc:260] T 12531195252640d18c3738a2c8053e61 P 51c98447074e4bf6bdd333832855692b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:27.937652 30710 ts_tablet_manager.cc:1428] T 12531195252640d18c3738a2c8053e61 P 51c98447074e4bf6bdd333832855692b: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260929 20:16:28.140852 30700 raft_consensus.cc:482] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:16:28.141075 30700 raft_consensus.cc:504] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:28.141832 30700 leader_election.cc:288] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51c98447074e4bf6bdd333832855692b (127.27.185.66:46807), 35549558d566453f8de25d956918d261 (127.27.185.67:40545)
I20260929 20:16:28.142493 30445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12531195252640d18c3738a2c8053e61" candidate_uuid: "058bb4d2a4d8493c8a15a926674d3a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51c98447074e4bf6bdd333832855692b" is_pre_election: true
I20260929 20:16:28.142553 30584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12531195252640d18c3738a2c8053e61" candidate_uuid: "058bb4d2a4d8493c8a15a926674d3a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35549558d566453f8de25d956918d261" is_pre_election: true
I20260929 20:16:28.142786 30445 raft_consensus.cc:2454] T 12531195252640d18c3738a2c8053e61 P 51c98447074e4bf6bdd333832855692b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 058bb4d2a4d8493c8a15a926674d3a73 in term 0.
I20260929 20:16:28.142902 30584 raft_consensus.cc:2454] T 12531195252640d18c3738a2c8053e61 P 35549558d566453f8de25d956918d261 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 058bb4d2a4d8493c8a15a926674d3a73 in term 0.
I20260929 20:16:28.143347 30236 leader_election.cc:302] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 058bb4d2a4d8493c8a15a926674d3a73, 51c98447074e4bf6bdd333832855692b; no voters:
I20260929 20:16:28.143679 30700 raft_consensus.cc:2781] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:16:28.143806 30700 raft_consensus.cc:482] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:16:28.143898 30700 raft_consensus.cc:3037] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:28.146209 30700 raft_consensus.cc:504] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:28.146773 30700 leader_election.cc:288] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73 [CANDIDATE]: Term 1 election: Requested vote from peers 51c98447074e4bf6bdd333832855692b (127.27.185.66:46807), 35549558d566453f8de25d956918d261 (127.27.185.67:40545)
I20260929 20:16:28.147313 30584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12531195252640d18c3738a2c8053e61" candidate_uuid: "058bb4d2a4d8493c8a15a926674d3a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35549558d566453f8de25d956918d261"
I20260929 20:16:28.147308 30445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12531195252640d18c3738a2c8053e61" candidate_uuid: "058bb4d2a4d8493c8a15a926674d3a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51c98447074e4bf6bdd333832855692b"
I20260929 20:16:28.147593 30584 raft_consensus.cc:3037] T 12531195252640d18c3738a2c8053e61 P 35549558d566453f8de25d956918d261 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:28.147593 30445 raft_consensus.cc:3037] T 12531195252640d18c3738a2c8053e61 P 51c98447074e4bf6bdd333832855692b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:28.149858 30584 raft_consensus.cc:2454] T 12531195252640d18c3738a2c8053e61 P 35549558d566453f8de25d956918d261 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 058bb4d2a4d8493c8a15a926674d3a73 in term 1.
I20260929 20:16:28.150309 30237 leader_election.cc:302] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 058bb4d2a4d8493c8a15a926674d3a73, 35549558d566453f8de25d956918d261; no voters:
I20260929 20:16:28.150637 30700 raft_consensus.cc:2781] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:16:28.150638 30445 raft_consensus.cc:2454] T 12531195252640d18c3738a2c8053e61 P 51c98447074e4bf6bdd333832855692b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 058bb4d2a4d8493c8a15a926674d3a73 in term 1.
I20260929 20:16:28.151165 30700 raft_consensus.cc:686] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73 [term 1 LEADER]: Becoming Leader. State: Replica: 058bb4d2a4d8493c8a15a926674d3a73, State: Running, Role: LEADER
I20260929 20:16:28.151520 30700 consensus_queue.cc:237] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:28.154536 30145 catalog_manager.cc:5495] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 058bb4d2a4d8493c8a15a926674d3a73 (127.27.185.65). New cstate: current_term: 1 leader_uuid: "058bb4d2a4d8493c8a15a926674d3a73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } health_report { overall_health: UNKNOWN } } }
I20260929 20:16:28.186319 30652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:28.181405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56350 (local address 127.27.185.65:38361)
0929 20:16:28.181587 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:28.181592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:28.181706 (+ 114us) server_negotiation.cc:407] Connection header received
0929 20:16:28.181916 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:28.181922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:28.181998 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:28.182107 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:16:28.182479 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:28.183200 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:28.184571 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:28.184981 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:28.185123 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:28.185546 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:16:28.185681 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:16:28.185781 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:28.185946 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 20:16:28.186030 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:16:28.199191 30652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:28.194804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:35983 (local address 127.27.185.65:38361)
0929 20:16:28.194951 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:28.194959 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:28.194977 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:16:28.195199 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:28.195203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:28.195254 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:28.195338 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:16:28.195673 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:28.196371 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:28.197584 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:28.197983 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:28.198123 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:28.198458 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:16:28.198562 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:16:28.198656 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:28.198784 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 20:16:28.198858 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":59}
I20260929 20:16:28.199191 30702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:28.194719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:38361 (local address 127.27.185.65:35983)
0929 20:16:28.194903 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:28.194923 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:28.195041 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:28.195348 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:28.195353 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:28.195363 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:16:28.195553 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:28.195561 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:28.196497 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:28.196512 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:28.197438 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:28.197449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:28.198140 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:28.198146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:28.198277 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:28.198309 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:16:28.198655 (+ 346us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:16:28.198666 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 20:16:28.198766 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 20:16:28.198896 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 20:16:28.209645 30584 raft_consensus.cc:1261] T 12531195252640d18c3738a2c8053e61 P 35549558d566453f8de25d956918d261 [term 1 FOLLOWER]: Refusing update from remote peer 058bb4d2a4d8493c8a15a926674d3a73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:16:28.209702 30445 raft_consensus.cc:1261] T 12531195252640d18c3738a2c8053e61 P 51c98447074e4bf6bdd333832855692b [term 1 FOLLOWER]: Refusing update from remote peer 058bb4d2a4d8493c8a15a926674d3a73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:16:28.210357 30700 consensus_queue.cc:1035] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:16:28.210649 30678 consensus_queue.cc:1035] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 20:16:28.233901 30687 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 20:16:29.823464 30145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:16:29.823698 30145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:16:29.827656 30145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34f1fba2fa284ab08e77a76867895f77: Sending DeleteTablet for 3 replicas of tablet 12531195252640d18c3738a2c8053e61
I20260929 20:16:29.828480 30425 tablet_service.cc:1514] Processing DeleteTablet for tablet 12531195252640d18c3738a2c8053e61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:16:29 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:50900
I20260929 20:16:29.828531 30564 tablet_service.cc:1514] Processing DeleteTablet for tablet 12531195252640d18c3738a2c8053e61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:16:29 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:60838
I20260929 20:16:29.828598 30284 tablet_service.cc:1514] Processing DeleteTablet for tablet 12531195252640d18c3738a2c8053e61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:16:29 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:56330
I20260929 20:16:29.845151 30740 tablet_replica.cc:330] T 12531195252640d18c3738a2c8053e61 P 51c98447074e4bf6bdd333832855692b: stopping tablet replica
I20260929 20:16:29.845440 30741 tablet_replica.cc:330] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73: stopping tablet replica
I20260929 20:16:29.845499 30740 raft_consensus.cc:2229] T 12531195252640d18c3738a2c8053e61 P 51c98447074e4bf6bdd333832855692b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:16:29.845733 30741 raft_consensus.cc:2229] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:16:29.846009 30740 raft_consensus.cc:2258] T 12531195252640d18c3738a2c8053e61 P 51c98447074e4bf6bdd333832855692b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:16:29.846452 30741 raft_consensus.cc:2258] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:16:29.847096 30739 tablet_replica.cc:330] T 12531195252640d18c3738a2c8053e61 P 35549558d566453f8de25d956918d261: stopping tablet replica
I20260929 20:16:29.847311 30739 raft_consensus.cc:2229] T 12531195252640d18c3738a2c8053e61 P 35549558d566453f8de25d956918d261 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:16:29.847338 30740 ts_tablet_manager.cc:1905] T 12531195252640d18c3738a2c8053e61 P 51c98447074e4bf6bdd333832855692b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:16:29.847586 30739 raft_consensus.cc:2258] T 12531195252640d18c3738a2c8053e61 P 35549558d566453f8de25d956918d261 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:16:29.848407 30741 ts_tablet_manager.cc:1905] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:16:29.848604 30739 ts_tablet_manager.cc:1905] T 12531195252640d18c3738a2c8053e61 P 35549558d566453f8de25d956918d261: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:16:29.855119 30741 ts_tablet_manager.cc:1918] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:16:29.855314 30741 log.cc:1196] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-0/wal/wals/12531195252640d18c3738a2c8053e61
I20260929 20:16:29.855688 30740 ts_tablet_manager.cc:1918] T 12531195252640d18c3738a2c8053e61 P 51c98447074e4bf6bdd333832855692b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:16:29.855732 30739 ts_tablet_manager.cc:1918] T 12531195252640d18c3738a2c8053e61 P 35549558d566453f8de25d956918d261: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:16:29.855855 30739 log.cc:1196] T 12531195252640d18c3738a2c8053e61 P 35549558d566453f8de25d956918d261: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-2/wal/wals/12531195252640d18c3738a2c8053e61
I20260929 20:16:29.855868 30741 ts_tablet_manager.cc:1939] T 12531195252640d18c3738a2c8053e61 P 058bb4d2a4d8493c8a15a926674d3a73: Deleting consensus metadata
I20260929 20:16:29.855913 30740 log.cc:1196] T 12531195252640d18c3738a2c8053e61 P 51c98447074e4bf6bdd333832855692b: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-1/wal/wals/12531195252640d18c3738a2c8053e61
I20260929 20:16:29.856351 30739 ts_tablet_manager.cc:1939] T 12531195252640d18c3738a2c8053e61 P 35549558d566453f8de25d956918d261: Deleting consensus metadata
I20260929 20:16:29.856482 30740 ts_tablet_manager.cc:1939] T 12531195252640d18c3738a2c8053e61 P 51c98447074e4bf6bdd333832855692b: Deleting consensus metadata
I20260929 20:16:29.857069 30132 catalog_manager.cc:4841] TS 058bb4d2a4d8493c8a15a926674d3a73 (127.27.185.65:38361): tablet 12531195252640d18c3738a2c8053e61 (table test-table [id=cb8be02b0f6c483096114695ebd778d5]) successfully deleted
I20260929 20:16:29.857733 30131 catalog_manager.cc:4841] TS 51c98447074e4bf6bdd333832855692b (127.27.185.66:46807): tablet 12531195252640d18c3738a2c8053e61 (table test-table [id=cb8be02b0f6c483096114695ebd778d5]) successfully deleted
I20260929 20:16:29.857836 30132 catalog_manager.cc:4841] TS 35549558d566453f8de25d956918d261 (127.27.185.67:40545): tablet 12531195252640d18c3738a2c8053e61 (table test-table [id=cb8be02b0f6c483096114695ebd778d5]) successfully deleted
Sep 29 20:16:29 dist-test-slave-4cx3 krb5kdc[30100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712989, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 20:16:29 dist-test-slave-4cx3 krb5kdc[30100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712989, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:29.894007 30749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:29.876258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55064 (local address 127.27.185.126:45305)
0929 20:16:29.876909 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:29.876915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:29.876942 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:16:29.877108 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:29.877114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:29.877202 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:29.877322 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:29.878720 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:29.879574 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:29.880821 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:29.881245 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:29.881531 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:29.888695 (+ 7164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:29.888701 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:29.888705 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:29.888741 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:29.891582 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:29.892245 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:29.892249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:29.892252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:29.892371 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:29.892736 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:29.892739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:29.892741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:29.893020 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:29.893189 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:29.893447 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 20:16:29.893617 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":323,"threads_started":1}
I20260929 20:16:29.897794 30145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:16:29.898450 30145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:16:29.905248 30284 tablet_service.cc:1467] Processing CreateTablet for tablet 006afe5b68e0464785852a89d0d95f40 (DEFAULT_TABLE table=test-table [id=093611c37f7544f2b22aacc16e2c96f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:16:29.905398 30425 tablet_service.cc:1467] Processing CreateTablet for tablet 006afe5b68e0464785852a89d0d95f40 (DEFAULT_TABLE table=test-table [id=093611c37f7544f2b22aacc16e2c96f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:16:29.905799 30425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 006afe5b68e0464785852a89d0d95f40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:29.905784 30284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 006afe5b68e0464785852a89d0d95f40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:29.905799 30564 tablet_service.cc:1467] Processing CreateTablet for tablet 006afe5b68e0464785852a89d0d95f40 (DEFAULT_TABLE table=test-table [id=093611c37f7544f2b22aacc16e2c96f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:16:29.906225 30564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 006afe5b68e0464785852a89d0d95f40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:29.911664 30750 tablet_bootstrap.cc:492] T 006afe5b68e0464785852a89d0d95f40 P 51c98447074e4bf6bdd333832855692b: Bootstrap starting.
I20260929 20:16:29.912708 30751 tablet_bootstrap.cc:492] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73: Bootstrap starting.
I20260929 20:16:29.913537 30752 tablet_bootstrap.cc:492] T 006afe5b68e0464785852a89d0d95f40 P 35549558d566453f8de25d956918d261: Bootstrap starting.
I20260929 20:16:29.914162 30750 tablet_bootstrap.cc:654] T 006afe5b68e0464785852a89d0d95f40 P 51c98447074e4bf6bdd333832855692b: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:29.914510 30751 tablet_bootstrap.cc:654] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:29.915293 30752 tablet_bootstrap.cc:654] T 006afe5b68e0464785852a89d0d95f40 P 35549558d566453f8de25d956918d261: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:29.916623 30750 tablet_bootstrap.cc:492] T 006afe5b68e0464785852a89d0d95f40 P 51c98447074e4bf6bdd333832855692b: No bootstrap required, opened a new log
I20260929 20:16:29.916780 30750 ts_tablet_manager.cc:1397] T 006afe5b68e0464785852a89d0d95f40 P 51c98447074e4bf6bdd333832855692b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 20:16:29.916818 30751 tablet_bootstrap.cc:492] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73: No bootstrap required, opened a new log
I20260929 20:16:29.916978 30751 ts_tablet_manager.cc:1397] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260929 20:16:29.917579 30752 tablet_bootstrap.cc:492] T 006afe5b68e0464785852a89d0d95f40 P 35549558d566453f8de25d956918d261: No bootstrap required, opened a new log
I20260929 20:16:29.917502 30750 raft_consensus.cc:348] T 006afe5b68e0464785852a89d0d95f40 P 51c98447074e4bf6bdd333832855692b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:29.917727 30750 raft_consensus.cc:374] T 006afe5b68e0464785852a89d0d95f40 P 51c98447074e4bf6bdd333832855692b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:29.917730 30752 ts_tablet_manager.cc:1397] T 006afe5b68e0464785852a89d0d95f40 P 35549558d566453f8de25d956918d261: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 20:16:29.917820 30750 raft_consensus.cc:729] T 006afe5b68e0464785852a89d0d95f40 P 51c98447074e4bf6bdd333832855692b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c98447074e4bf6bdd333832855692b, State: Initialized, Role: FOLLOWER
I20260929 20:16:29.918082 30750 consensus_queue.cc:260] T 006afe5b68e0464785852a89d0d95f40 P 51c98447074e4bf6bdd333832855692b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:29.917806 30751 raft_consensus.cc:348] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:29.918331 30751 raft_consensus.cc:374] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:29.918440 30751 raft_consensus.cc:729] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 058bb4d2a4d8493c8a15a926674d3a73, State: Initialized, Role: FOLLOWER
I20260929 20:16:29.918496 30752 raft_consensus.cc:348] T 006afe5b68e0464785852a89d0d95f40 P 35549558d566453f8de25d956918d261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:29.918824 30752 raft_consensus.cc:374] T 006afe5b68e0464785852a89d0d95f40 P 35549558d566453f8de25d956918d261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:29.918959 30752 raft_consensus.cc:729] T 006afe5b68e0464785852a89d0d95f40 P 35549558d566453f8de25d956918d261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35549558d566453f8de25d956918d261, State: Initialized, Role: FOLLOWER
I20260929 20:16:29.918923 30751 consensus_queue.cc:260] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:29.919240 30752 consensus_queue.cc:260] T 006afe5b68e0464785852a89d0d95f40 P 35549558d566453f8de25d956918d261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:29.920050 30751 ts_tablet_manager.cc:1428] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260929 20:16:29.920881 30752 ts_tablet_manager.cc:1428] T 006afe5b68e0464785852a89d0d95f40 P 35549558d566453f8de25d956918d261: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 20:16:29.927963 30750 ts_tablet_manager.cc:1428] T 006afe5b68e0464785852a89d0d95f40 P 51c98447074e4bf6bdd333832855692b: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260929 20:16:30.027453 30700 raft_consensus.cc:482] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:16:30.027707 30700 raft_consensus.cc:504] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:30.028465 30700 leader_election.cc:288] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51c98447074e4bf6bdd333832855692b (127.27.185.66:46807), 35549558d566453f8de25d956918d261 (127.27.185.67:40545)
I20260929 20:16:30.029098 30445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006afe5b68e0464785852a89d0d95f40" candidate_uuid: "058bb4d2a4d8493c8a15a926674d3a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51c98447074e4bf6bdd333832855692b" is_pre_election: true
I20260929 20:16:30.029126 30584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006afe5b68e0464785852a89d0d95f40" candidate_uuid: "058bb4d2a4d8493c8a15a926674d3a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35549558d566453f8de25d956918d261" is_pre_election: true
I20260929 20:16:30.029402 30445 raft_consensus.cc:2454] T 006afe5b68e0464785852a89d0d95f40 P 51c98447074e4bf6bdd333832855692b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 058bb4d2a4d8493c8a15a926674d3a73 in term 0.
I20260929 20:16:30.029387 30584 raft_consensus.cc:2454] T 006afe5b68e0464785852a89d0d95f40 P 35549558d566453f8de25d956918d261 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 058bb4d2a4d8493c8a15a926674d3a73 in term 0.
I20260929 20:16:30.029914 30237 leader_election.cc:302] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 058bb4d2a4d8493c8a15a926674d3a73, 35549558d566453f8de25d956918d261, 51c98447074e4bf6bdd333832855692b; no voters:
I20260929 20:16:30.030313 30700 raft_consensus.cc:2781] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:16:30.030462 30700 raft_consensus.cc:482] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:16:30.030576 30700 raft_consensus.cc:3037] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:30.032771 30700 raft_consensus.cc:504] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:30.033480 30700 leader_election.cc:288] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73 [CANDIDATE]: Term 1 election: Requested vote from peers 51c98447074e4bf6bdd333832855692b (127.27.185.66:46807), 35549558d566453f8de25d956918d261 (127.27.185.67:40545)
I20260929 20:16:30.033937 30445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006afe5b68e0464785852a89d0d95f40" candidate_uuid: "058bb4d2a4d8493c8a15a926674d3a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51c98447074e4bf6bdd333832855692b"
I20260929 20:16:30.034083 30584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006afe5b68e0464785852a89d0d95f40" candidate_uuid: "058bb4d2a4d8493c8a15a926674d3a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35549558d566453f8de25d956918d261"
I20260929 20:16:30.034200 30445 raft_consensus.cc:3037] T 006afe5b68e0464785852a89d0d95f40 P 51c98447074e4bf6bdd333832855692b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:30.034329 30584 raft_consensus.cc:3037] T 006afe5b68e0464785852a89d0d95f40 P 35549558d566453f8de25d956918d261 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:30.036319 30445 raft_consensus.cc:2454] T 006afe5b68e0464785852a89d0d95f40 P 51c98447074e4bf6bdd333832855692b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 058bb4d2a4d8493c8a15a926674d3a73 in term 1.
I20260929 20:16:30.036767 30236 leader_election.cc:302] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 058bb4d2a4d8493c8a15a926674d3a73, 51c98447074e4bf6bdd333832855692b; no voters:
I20260929 20:16:30.037068 30700 raft_consensus.cc:2781] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:16:30.037235 30700 raft_consensus.cc:686] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73 [term 1 LEADER]: Becoming Leader. State: Replica: 058bb4d2a4d8493c8a15a926674d3a73, State: Running, Role: LEADER
I20260929 20:16:30.037390 30584 raft_consensus.cc:2454] T 006afe5b68e0464785852a89d0d95f40 P 35549558d566453f8de25d956918d261 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 058bb4d2a4d8493c8a15a926674d3a73 in term 1.
I20260929 20:16:30.037540 30700 consensus_queue.cc:237] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:30.040446 30145 catalog_manager.cc:5495] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 058bb4d2a4d8493c8a15a926674d3a73 (127.27.185.65). New cstate: current_term: 1 leader_uuid: "058bb4d2a4d8493c8a15a926674d3a73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } health_report { overall_health: UNKNOWN } } }
I20260929 20:16:30.079401 30761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:30.073901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56352 (local address 127.27.185.65:38361)
0929 20:16:30.074547 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:30.074554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:30.074573 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:16:30.074636 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:30.074639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:30.074698 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:30.074822 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:16:30.075260 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:30.076135 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:30.077469 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:30.077956 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:30.078097 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:30.078588 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:16:30.078701 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:16:30.078805 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:30.078973 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 20:16:30.079053 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":354,"threads_started":1}
I20260929 20:16:30.089167 30445 raft_consensus.cc:1261] T 006afe5b68e0464785852a89d0d95f40 P 51c98447074e4bf6bdd333832855692b [term 1 FOLLOWER]: Refusing update from remote peer 058bb4d2a4d8493c8a15a926674d3a73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:16:30.089205 30584 raft_consensus.cc:1261] T 006afe5b68e0464785852a89d0d95f40 P 35549558d566453f8de25d956918d261 [term 1 FOLLOWER]: Refusing update from remote peer 058bb4d2a4d8493c8a15a926674d3a73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:16:30.089792 30759 consensus_queue.cc:1035] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:16:30.090145 30722 consensus_queue.cc:1035] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:16:30.133666 30145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:16:30.133939 30145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:16:30.136953 30145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34f1fba2fa284ab08e77a76867895f77: Sending DeleteTablet for 3 replicas of tablet 006afe5b68e0464785852a89d0d95f40
I20260929 20:16:30.137907 30425 tablet_service.cc:1514] Processing DeleteTablet for tablet 006afe5b68e0464785852a89d0d95f40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:16:30 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:50900
I20260929 20:16:30.137910 30564 tablet_service.cc:1514] Processing DeleteTablet for tablet 006afe5b68e0464785852a89d0d95f40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:16:30 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:60838
I20260929 20:16:30.137933 30285 tablet_service.cc:1514] Processing DeleteTablet for tablet 006afe5b68e0464785852a89d0d95f40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:16:30 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:56330
I20260929 20:16:30.138219 30741 tablet_replica.cc:330] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73: stopping tablet replica
I20260929 20:16:30.138231 30739 tablet_replica.cc:330] T 006afe5b68e0464785852a89d0d95f40 P 35549558d566453f8de25d956918d261: stopping tablet replica
I20260929 20:16:30.138252 30740 tablet_replica.cc:330] T 006afe5b68e0464785852a89d0d95f40 P 51c98447074e4bf6bdd333832855692b: stopping tablet replica
I20260929 20:16:30.138423 30741 raft_consensus.cc:2229] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:16:30.138429 30739 raft_consensus.cc:2229] T 006afe5b68e0464785852a89d0d95f40 P 35549558d566453f8de25d956918d261 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:16:30.138486 30740 raft_consensus.cc:2229] T 006afe5b68e0464785852a89d0d95f40 P 51c98447074e4bf6bdd333832855692b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:16:30.138623 30739 raft_consensus.cc:2258] T 006afe5b68e0464785852a89d0d95f40 P 35549558d566453f8de25d956918d261 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:16:30.138705 30740 raft_consensus.cc:2258] T 006afe5b68e0464785852a89d0d95f40 P 51c98447074e4bf6bdd333832855692b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:16:30.138749 30741 raft_consensus.cc:2258] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:16:30.139005 30145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:16:30.139896 30145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:16:30.139933 30741 ts_tablet_manager.cc:1905] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:16:30.139971 30740 ts_tablet_manager.cc:1905] T 006afe5b68e0464785852a89d0d95f40 P 51c98447074e4bf6bdd333832855692b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:16:30.141811 30739 ts_tablet_manager.cc:1905] T 006afe5b68e0464785852a89d0d95f40 P 35549558d566453f8de25d956918d261: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:16:30.145437 30741 ts_tablet_manager.cc:1918] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:16:30.145622 30741 log.cc:1196] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-0/wal/wals/006afe5b68e0464785852a89d0d95f40
I20260929 20:16:30.146000 30741 ts_tablet_manager.cc:1939] T 006afe5b68e0464785852a89d0d95f40 P 058bb4d2a4d8493c8a15a926674d3a73: Deleting consensus metadata
I20260929 20:16:30.146944 30132 catalog_manager.cc:4841] TS 058bb4d2a4d8493c8a15a926674d3a73 (127.27.185.65:38361): tablet 006afe5b68e0464785852a89d0d95f40 (table test-table [id=093611c37f7544f2b22aacc16e2c96f8]) successfully deleted
I20260929 20:16:30.147544 30740 ts_tablet_manager.cc:1918] T 006afe5b68e0464785852a89d0d95f40 P 51c98447074e4bf6bdd333832855692b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:16:30.147696 30740 log.cc:1196] T 006afe5b68e0464785852a89d0d95f40 P 51c98447074e4bf6bdd333832855692b: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-1/wal/wals/006afe5b68e0464785852a89d0d95f40
I20260929 20:16:30.148105 30740 ts_tablet_manager.cc:1939] T 006afe5b68e0464785852a89d0d95f40 P 51c98447074e4bf6bdd333832855692b: Deleting consensus metadata
I20260929 20:16:30.149142 30739 ts_tablet_manager.cc:1918] T 006afe5b68e0464785852a89d0d95f40 P 35549558d566453f8de25d956918d261: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:16:30.149142 30131 catalog_manager.cc:4841] TS 51c98447074e4bf6bdd333832855692b (127.27.185.66:46807): tablet 006afe5b68e0464785852a89d0d95f40 (table test-table [id=093611c37f7544f2b22aacc16e2c96f8]) successfully deleted
I20260929 20:16:30.149317 30739 log.cc:1196] T 006afe5b68e0464785852a89d0d95f40 P 35549558d566453f8de25d956918d261: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-2/wal/wals/006afe5b68e0464785852a89d0d95f40
I20260929 20:16:30.149761 30739 ts_tablet_manager.cc:1939] T 006afe5b68e0464785852a89d0d95f40 P 35549558d566453f8de25d956918d261: Deleting consensus metadata
I20260929 20:16:30.150774 30132 catalog_manager.cc:4841] TS 35549558d566453f8de25d956918d261 (127.27.185.67:40545): tablet 006afe5b68e0464785852a89d0d95f40 (table test-table [id=093611c37f7544f2b22aacc16e2c96f8]) successfully deleted
I20260929 20:16:30.151680 30285 tablet_service.cc:1467] Processing CreateTablet for tablet bed8705795394061b69b69a425f2578e (DEFAULT_TABLE table=test-table [id=b1fdc21ad0144fb1ae94d85994b45d5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:16:30.152119 30285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bed8705795394061b69b69a425f2578e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:30.152195 30564 tablet_service.cc:1467] Processing CreateTablet for tablet bed8705795394061b69b69a425f2578e (DEFAULT_TABLE table=test-table [id=b1fdc21ad0144fb1ae94d85994b45d5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:16:30.152177 30425 tablet_service.cc:1467] Processing CreateTablet for tablet bed8705795394061b69b69a425f2578e (DEFAULT_TABLE table=test-table [id=b1fdc21ad0144fb1ae94d85994b45d5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:16:30.152576 30564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bed8705795394061b69b69a425f2578e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:30.152597 30425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bed8705795394061b69b69a425f2578e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:30.156384 30751 tablet_bootstrap.cc:492] T bed8705795394061b69b69a425f2578e P 058bb4d2a4d8493c8a15a926674d3a73: Bootstrap starting.
I20260929 20:16:30.157740 30751 tablet_bootstrap.cc:654] T bed8705795394061b69b69a425f2578e P 058bb4d2a4d8493c8a15a926674d3a73: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:30.159019 30752 tablet_bootstrap.cc:492] T bed8705795394061b69b69a425f2578e P 35549558d566453f8de25d956918d261: Bootstrap starting.
I20260929 20:16:30.159740 30750 tablet_bootstrap.cc:492] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b: Bootstrap starting.
I20260929 20:16:30.159971 30751 tablet_bootstrap.cc:492] T bed8705795394061b69b69a425f2578e P 058bb4d2a4d8493c8a15a926674d3a73: No bootstrap required, opened a new log
I20260929 20:16:30.160164 30751 ts_tablet_manager.cc:1397] T bed8705795394061b69b69a425f2578e P 058bb4d2a4d8493c8a15a926674d3a73: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260929 20:16:30.160718 30752 tablet_bootstrap.cc:654] T bed8705795394061b69b69a425f2578e P 35549558d566453f8de25d956918d261: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:30.160910 30751 raft_consensus.cc:348] T bed8705795394061b69b69a425f2578e P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:30.161137 30751 raft_consensus.cc:374] T bed8705795394061b69b69a425f2578e P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:30.161221 30751 raft_consensus.cc:729] T bed8705795394061b69b69a425f2578e P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 058bb4d2a4d8493c8a15a926674d3a73, State: Initialized, Role: FOLLOWER
I20260929 20:16:30.161420 30750 tablet_bootstrap.cc:654] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:30.161516 30751 consensus_queue.cc:260] T bed8705795394061b69b69a425f2578e P 058bb4d2a4d8493c8a15a926674d3a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:30.162067 30751 ts_tablet_manager.cc:1428] T bed8705795394061b69b69a425f2578e P 058bb4d2a4d8493c8a15a926674d3a73: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 20:16:30.163594 30752 tablet_bootstrap.cc:492] T bed8705795394061b69b69a425f2578e P 35549558d566453f8de25d956918d261: No bootstrap required, opened a new log
I20260929 20:16:30.163798 30752 ts_tablet_manager.cc:1397] T bed8705795394061b69b69a425f2578e P 35549558d566453f8de25d956918d261: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 20:16:30.164423 30752 raft_consensus.cc:348] T bed8705795394061b69b69a425f2578e P 35549558d566453f8de25d956918d261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:30.164685 30752 raft_consensus.cc:374] T bed8705795394061b69b69a425f2578e P 35549558d566453f8de25d956918d261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:30.164768 30752 raft_consensus.cc:729] T bed8705795394061b69b69a425f2578e P 35549558d566453f8de25d956918d261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35549558d566453f8de25d956918d261, State: Initialized, Role: FOLLOWER
I20260929 20:16:30.165015 30752 consensus_queue.cc:260] T bed8705795394061b69b69a425f2578e P 35549558d566453f8de25d956918d261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:30.165547 30752 ts_tablet_manager.cc:1428] T bed8705795394061b69b69a425f2578e P 35549558d566453f8de25d956918d261: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:16:30.166823 30750 tablet_bootstrap.cc:492] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b: No bootstrap required, opened a new log
I20260929 20:16:30.167025 30750 ts_tablet_manager.cc:1397] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20260929 20:16:30.167865 30750 raft_consensus.cc:348] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:30.168119 30750 raft_consensus.cc:374] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:30.168217 30750 raft_consensus.cc:729] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c98447074e4bf6bdd333832855692b, State: Initialized, Role: FOLLOWER
I20260929 20:16:30.168439 30750 consensus_queue.cc:260] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:30.168890 30750 ts_tablet_manager.cc:1428] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 20:16:30.224152 30756 raft_consensus.cc:482] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:16:30.224402 30756 raft_consensus.cc:504] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:30.225976 30756 leader_election.cc:288] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 058bb4d2a4d8493c8a15a926674d3a73 (127.27.185.65:38361), 35549558d566453f8de25d956918d261 (127.27.185.67:40545)
I20260929 20:16:30.233747 30761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:30.226554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:52001 (local address 127.27.185.65:38361)
0929 20:16:30.226776 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:30.226784 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:30.227032 (+ 248us) server_negotiation.cc:407] Connection header received
0929 20:16:30.227263 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:30.227271 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:30.227363 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:30.227480 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:16:30.228166 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:30.229233 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:30.231649 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:30.232829 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:30.232997 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:30.233073 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:30.233292 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 20:16:30.233429 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:16:30.233839 30768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:30.226260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:38361 (local address 127.27.185.66:52001)
0929 20:16:30.226909 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:30.226932 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:30.227109 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:30.227653 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:30.227657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:30.227681 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:16:30.228001 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:30.228010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:30.229378 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:30.229381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:30.231500 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:30.231520 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:30.233021 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:30.233025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:30.233131 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:30.233156 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 20:16:30.233234 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 20:16:30.233392 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":309,"threads_started":1}
I20260929 20:16:30.234334 30769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:30.226916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:40545 (local address 127.27.185.66:54159)
0929 20:16:30.227722 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:30.227741 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:30.227844 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:30.228293 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:30.228296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:30.228315 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:16:30.228586 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:30.228592 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:30.229851 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:30.229857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:30.231808 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:30.231817 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:30.233435 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:30.233448 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:30.233630 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:30.233664 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 20:16:30.233842 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 20:16:30.233990 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":444,"threads_started":1}
I20260929 20:16:30.234377 30770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:30.227147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:54159 (local address 127.27.185.67:40545)
0929 20:16:30.227964 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:30.227969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:30.227990 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:16:30.228055 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:30.228059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:30.228166 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:30.228278 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:16:30.228745 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:30.229728 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:30.231970 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:30.233201 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:30.233467 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:30.233536 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:30.233884 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 20:16:30.234050 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":575,"threads_started":1}
I20260929 20:16:30.234833 30305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed8705795394061b69b69a425f2578e" candidate_uuid: "51c98447074e4bf6bdd333832855692b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "058bb4d2a4d8493c8a15a926674d3a73" is_pre_election: true
I20260929 20:16:30.235296 30305 raft_consensus.cc:2454] T bed8705795394061b69b69a425f2578e P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51c98447074e4bf6bdd333832855692b in term 0.
I20260929 20:16:30.235186 30585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed8705795394061b69b69a425f2578e" candidate_uuid: "51c98447074e4bf6bdd333832855692b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35549558d566453f8de25d956918d261" is_pre_election: true
I20260929 20:16:30.235510 30585 raft_consensus.cc:2454] T bed8705795394061b69b69a425f2578e P 35549558d566453f8de25d956918d261 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51c98447074e4bf6bdd333832855692b in term 0.
I20260929 20:16:30.235874 30379 leader_election.cc:302] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 058bb4d2a4d8493c8a15a926674d3a73, 51c98447074e4bf6bdd333832855692b; no voters:
I20260929 20:16:30.236253 30756 raft_consensus.cc:2781] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:16:30.236428 30756 raft_consensus.cc:482] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:16:30.236536 30756 raft_consensus.cc:3037] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:30.238833 30756 raft_consensus.cc:504] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:30.239459 30756 leader_election.cc:288] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b [CANDIDATE]: Term 1 election: Requested vote from peers 058bb4d2a4d8493c8a15a926674d3a73 (127.27.185.65:38361), 35549558d566453f8de25d956918d261 (127.27.185.67:40545)
I20260929 20:16:30.239907 30305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed8705795394061b69b69a425f2578e" candidate_uuid: "51c98447074e4bf6bdd333832855692b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "058bb4d2a4d8493c8a15a926674d3a73"
I20260929 20:16:30.240005 30585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed8705795394061b69b69a425f2578e" candidate_uuid: "51c98447074e4bf6bdd333832855692b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35549558d566453f8de25d956918d261"
I20260929 20:16:30.240128 30305 raft_consensus.cc:3037] T bed8705795394061b69b69a425f2578e P 058bb4d2a4d8493c8a15a926674d3a73 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:30.240243 30585 raft_consensus.cc:3037] T bed8705795394061b69b69a425f2578e P 35549558d566453f8de25d956918d261 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:30.242252 30305 raft_consensus.cc:2454] T bed8705795394061b69b69a425f2578e P 058bb4d2a4d8493c8a15a926674d3a73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51c98447074e4bf6bdd333832855692b in term 1.
I20260929 20:16:30.242429 30585 raft_consensus.cc:2454] T bed8705795394061b69b69a425f2578e P 35549558d566453f8de25d956918d261 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51c98447074e4bf6bdd333832855692b in term 1.
I20260929 20:16:30.242678 30379 leader_election.cc:302] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 058bb4d2a4d8493c8a15a926674d3a73, 51c98447074e4bf6bdd333832855692b; no voters:
I20260929 20:16:30.242977 30756 raft_consensus.cc:2781] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:16:30.243459 30756 raft_consensus.cc:686] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b [term 1 LEADER]: Becoming Leader. State: Replica: 51c98447074e4bf6bdd333832855692b, State: Running, Role: LEADER
I20260929 20:16:30.243901 30756 consensus_queue.cc:237] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } }
I20260929 20:16:30.247258 30143 catalog_manager.cc:5495] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b reported cstate change: term changed from 0 to 1, leader changed from <none> to 51c98447074e4bf6bdd333832855692b (127.27.185.66). New cstate: current_term: 1 leader_uuid: "51c98447074e4bf6bdd333832855692b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51c98447074e4bf6bdd333832855692b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 46807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 } health_report { overall_health: UNKNOWN } } }
I20260929 20:16:30.276780 30773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:30.270545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50910 (local address 127.27.185.66:46807)
0929 20:16:30.271150 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:30.271154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:30.271174 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:16:30.271255 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:30.271259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:30.271335 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:30.271544 (+ 209us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:16:30.271915 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:30.272715 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:30.274105 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:30.274610 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:30.274711 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:30.275185 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:16:30.276014 (+ 829us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:16:30.276135 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:30.276415 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 20:16:30.276486 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":366,"threads_started":1}
I20260929 20:16:30.293452 30761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:30.287270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:45773 (local address 127.27.185.65:38361)
0929 20:16:30.287480 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:30.287486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:30.288124 (+ 638us) server_negotiation.cc:407] Connection header received
0929 20:16:30.288359 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:30.288363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:30.288471 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:30.288548 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:16:30.289085 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:30.289806 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:30.291801 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:30.292215 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:30.292310 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:30.292823 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:16:30.292923 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:16:30.292989 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:30.293143 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 20:16:30.293206 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:16:30.293635 30777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:30.287288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:38361 (local address 127.27.185.66:45773)
0929 20:16:30.288016 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:30.288035 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:30.288176 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:30.288592 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:30.288597 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:30.288611 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:16:30.288941 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:30.288952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:30.289944 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:30.289949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:30.291654 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:30.291669 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:30.292389 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:30.292396 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:30.292581 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:30.292650 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:16:30.293018 (+ 368us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:16:30.293030 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 20:16:30.293108 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 20:16:30.293256 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":354,"threads_started":1}
I20260929 20:16:30.302603 30585 raft_consensus.cc:1261] T bed8705795394061b69b69a425f2578e P 35549558d566453f8de25d956918d261 [term 1 FOLLOWER]: Refusing update from remote peer 51c98447074e4bf6bdd333832855692b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:16:30.302618 30305 raft_consensus.cc:1261] T bed8705795394061b69b69a425f2578e P 058bb4d2a4d8493c8a15a926674d3a73 [term 1 FOLLOWER]: Refusing update from remote peer 51c98447074e4bf6bdd333832855692b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:16:30.303325 30771 consensus_queue.cc:1035] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b [LEADER]: Connected to new peer: Peer: permanent_uuid: "35549558d566453f8de25d956918d261" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:16:30.304030 30756 consensus_queue.cc:1035] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b [LEADER]: Connected to new peer: Peer: permanent_uuid: "058bb4d2a4d8493c8a15a926674d3a73" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:16:30.330797 30773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:30.325193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:50707 (local address 127.27.185.66:46807)
0929 20:16:30.325362 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:30.325368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:30.325835 (+ 467us) server_negotiation.cc:407] Connection header received
0929 20:16:30.326033 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:30.326038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:30.326110 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:30.326181 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:16:30.326596 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:30.327507 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:30.329161 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:30.329588 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:30.329718 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:30.330125 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:16:30.330232 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:16:30.330329 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:30.330442 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 20:16:30.330516 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 20:16:30.330945 30782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:30.325087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:46807 (local address 127.27.185.65:50707)
0929 20:16:30.325749 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:30.325769 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:30.325912 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:30.326185 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:30.326187 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:30.326196 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:16:30.326465 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:30.326473 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:30.327642 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:30.327647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:30.329012 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:30.329026 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:30.329770 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:30.329779 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:30.329926 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:30.329962 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:16:30.330331 (+ 369us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:16:30.330345 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 20:16:30.330424 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 20:16:30.330555 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":358,"threads_started":1}
W20260929 20:16:30.332079 30725 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 20:16:31.921568 30143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:16:31.921911 30143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:16:31.928014 30143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34f1fba2fa284ab08e77a76867895f77: Sending DeleteTablet for 3 replicas of tablet bed8705795394061b69b69a425f2578e
I20260929 20:16:31.928848 30285 tablet_service.cc:1514] Processing DeleteTablet for tablet bed8705795394061b69b69a425f2578e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:16:31 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:56330
I20260929 20:16:31.928849 30425 tablet_service.cc:1514] Processing DeleteTablet for tablet bed8705795394061b69b69a425f2578e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:16:31 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:50900
I20260929 20:16:31.928937 30564 tablet_service.cc:1514] Processing DeleteTablet for tablet bed8705795394061b69b69a425f2578e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:16:31 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:60838
I20260929 20:16:31.929641 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 30218
I20260929 20:16:31.942152 30808 tablet_replica.cc:330] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b: stopping tablet replica
I20260929 20:16:31.942669 30808 raft_consensus.cc:2229] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:16:31.943477 30808 raft_consensus.cc:2258] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:16:31.944984 30808 ts_tablet_manager.cc:1905] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:16:31.954360 30808 ts_tablet_manager.cc:1918] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:16:31.954679 30808 log.cc:1196] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-1/wal/wals/bed8705795394061b69b69a425f2578e
I20260929 20:16:31.955180 30808 ts_tablet_manager.cc:1939] T bed8705795394061b69b69a425f2578e P 51c98447074e4bf6bdd333832855692b: Deleting consensus metadata
I20260929 20:16:31.956568 30131 catalog_manager.cc:4841] TS 51c98447074e4bf6bdd333832855692b (127.27.185.66:46807): tablet bed8705795394061b69b69a425f2578e (table test-table [id=b1fdc21ad0144fb1ae94d85994b45d5e]) successfully deleted
I20260929 20:16:31.968124 30809 tablet_replica.cc:330] T bed8705795394061b69b69a425f2578e P 058bb4d2a4d8493c8a15a926674d3a73: stopping tablet replica
I20260929 20:16:31.968425 30809 raft_consensus.cc:2229] T bed8705795394061b69b69a425f2578e P 058bb4d2a4d8493c8a15a926674d3a73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:16:31.968715 30809 raft_consensus.cc:2258] T bed8705795394061b69b69a425f2578e P 058bb4d2a4d8493c8a15a926674d3a73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:16:31.969381 30810 tablet_replica.cc:330] T bed8705795394061b69b69a425f2578e P 35549558d566453f8de25d956918d261: stopping tablet replica
I20260929 20:16:31.969611 30810 raft_consensus.cc:2229] T bed8705795394061b69b69a425f2578e P 35549558d566453f8de25d956918d261 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:16:31.969872 30810 raft_consensus.cc:2258] T bed8705795394061b69b69a425f2578e P 35549558d566453f8de25d956918d261 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:16:31.971226 30809 ts_tablet_manager.cc:1905] T bed8705795394061b69b69a425f2578e P 058bb4d2a4d8493c8a15a926674d3a73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:16:31.971292 30810 ts_tablet_manager.cc:1905] T bed8705795394061b69b69a425f2578e P 35549558d566453f8de25d956918d261: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:16:31.978420 30809 ts_tablet_manager.cc:1918] T bed8705795394061b69b69a425f2578e P 058bb4d2a4d8493c8a15a926674d3a73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:16:31.978601 30809 log.cc:1196] T bed8705795394061b69b69a425f2578e P 058bb4d2a4d8493c8a15a926674d3a73: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-0/wal/wals/bed8705795394061b69b69a425f2578e
I20260929 20:16:31.979120 30809 ts_tablet_manager.cc:1939] T bed8705795394061b69b69a425f2578e P 058bb4d2a4d8493c8a15a926674d3a73: Deleting consensus metadata
Sep 29 20:16:31 dist-test-slave-4cx3 krb5kdc[30100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712989, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.185.65@KRBTEST.COM
I20260929 20:16:31.980398 30132 catalog_manager.cc:4841] TS 058bb4d2a4d8493c8a15a926674d3a73 (127.27.185.65:38361): tablet bed8705795394061b69b69a425f2578e (table test-table [id=b1fdc21ad0144fb1ae94d85994b45d5e]) successfully deleted
I20260929 20:16:31.982124 30810 ts_tablet_manager.cc:1918] T bed8705795394061b69b69a425f2578e P 35549558d566453f8de25d956918d261: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:16:31.982278 30810 log.cc:1196] T bed8705795394061b69b69a425f2578e P 35549558d566453f8de25d956918d261: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790712966917858-28389-0/minicluster-data/ts-2/wal/wals/bed8705795394061b69b69a425f2578e
I20260929 20:16:31.982748 30810 ts_tablet_manager.cc:1939] T bed8705795394061b69b69a425f2578e P 35549558d566453f8de25d956918d261: Deleting consensus metadata
I20260929 20:16:31.984022 30132 catalog_manager.cc:4841] TS 35549558d566453f8de25d956918d261 (127.27.185.67:40545): tablet bed8705795394061b69b69a425f2578e (table test-table [id=b1fdc21ad0144fb1ae94d85994b45d5e]) successfully deleted
I20260929 20:16:31.993346 30812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:31.931716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56366 (local address 127.27.185.65:38361)
0929 20:16:31.958427 (+ 26711us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:31.958436 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:31.958468 (+ 32us) server_negotiation.cc:407] Connection header received
0929 20:16:31.958548 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:31.958571 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:31.958687 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:31.958825 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:31.961456 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:31.963117 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:31.965782 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:31.966777 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:31.966938 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:31.985050 (+ 18112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:31.985066 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:31.985128 (+ 62us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:31.985178 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:31.989510 (+ 4332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:31.990547 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:31.990556 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:31.990560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:31.990714 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:31.991199 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:31.991204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:31.991207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:31.991809 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:31.992140 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:31.992476 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 20:16:31.992725 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26529,"thread_start_us":31498,"threads_started":1}
W20260929 20:16:31.994285 30350 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56366
W20260929 20:16:31.994977 28389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 30218: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:16:31.995180 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 30218
I20260929 20:16:32.021368 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 30360
Sep 29 20:16:32 dist-test-slave-4cx3 krb5kdc[30100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712989, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.185.66@KRBTEST.COM
I20260929 20:16:32.039180 30813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:32.022119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50924 (local address 127.27.185.66:46807)
0929 20:16:32.022634 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:32.022638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:32.022654 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:16:32.022708 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:32.022711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:32.022769 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:32.022899 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:32.024530 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:32.025258 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:32.027059 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:32.027564 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:32.027672 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:32.032987 (+ 5315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:32.033010 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:32.033036 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:32.033084 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:32.036323 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:32.037135 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:32.037142 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:32.037144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:32.037272 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:32.037697 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:32.037701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:32.037703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:32.038087 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:32.038304 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:32.038661 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 20:16:32.038841 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":275,"threads_started":1}
W20260929 20:16:32.040120 30490 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50924
W20260929 20:16:32.040762 28389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 30360: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:16:32.040961 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 30360
I20260929 20:16:32.064023 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 30500
Sep 29 20:16:32 dist-test-slave-4cx3 krb5kdc[30100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712989, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.185.67@KRBTEST.COM
I20260929 20:16:32.080582 30814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:32.064841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60856 (local address 127.27.185.67:40545)
0929 20:16:32.065395 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:32.065400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:32.065416 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:16:32.065472 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:32.065475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:32.065551 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:32.065645 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:32.067046 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:32.067948 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:32.069219 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:32.069894 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:32.070018 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:32.074708 (+ 4690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:32.074726 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:32.074744 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:32.074785 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:32.077857 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:32.078558 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:32.078565 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:32.078567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:32.078701 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:32.079052 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:32.079055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:32.079057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:32.079519 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:32.079758 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:32.080015 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 20:16:32.080187 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":305,"threads_started":1}
W20260929 20:16:32.081332 30630 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60856
W20260929 20:16:32.082017 28389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 30500: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:16:32.082196 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 30500
I20260929 20:16:32.104620 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 30113
I20260929 20:16:32.105822 30186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:16:32.223337 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 30113
2026-09-29T20:16:32Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8652 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:16:32 dist-test-slave-4cx3 krb5kdc[30818](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:16:32 dist-test-slave-4cx3 krb5kdc[30818](info): set up 2 sockets
Sep 29 20:16:32 dist-test-slave-4cx3 krb5kdc[30818](info): commencing operation
krb5kdc: starting...
W20260929 20:16:33.318781 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:16:33 dist-test-slave-4cx3 krb5kdc[30818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:16:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:16:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
I20260929 20:16:33.514449 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:43481
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:33805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:43481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:33.838810 30831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:33.839178 30831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:33.839324 30831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:33.847086 30831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:16:33.847231 30831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:33.847321 30831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:33.847432 30831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:16:33.847502 30831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:16:33.857496 30831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:33805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:43481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:43481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:33.858855 30831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:33.860271 30831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:33.873380 30836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:33.873379 30837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:33.873379 30839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:33.873835 30831 server_base.cc:1034] running on GCE node
I20260929 20:16:33.874853 30831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:33.876399 30831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:33.877610 30831 hybrid_clock.cc:648] HybridClock initialized: now 1790712993877559 us; error 39 us; skew 500 ppm
I20260929 20:16:33.877993 30831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:33 dist-test-slave-4cx3 krb5kdc[30818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712993, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:33.881289 30831 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:16:33.883051 30831 webserver.cc:460] Webserver started at http://127.27.185.126:45557/ using document root <none> and password file <none>
I20260929 20:16:33.883962 30831 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:33.884110 30831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:33.884522 30831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:33.887766 30831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "458a6c4473b9465e9cdc1a7869524cd7"
format_stamp: "Formatted at 2026-09-29 20:16:33 on dist-test-slave-4cx3"
server_key: "e556814529080851aac1267c4c9b7157"
server_key_iv: "b4738734782d172e9a18fd62b98c2b48"
server_key_version: "encryptionkey@0"
I20260929 20:16:33.888540 30831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "458a6c4473b9465e9cdc1a7869524cd7"
format_stamp: "Formatted at 2026-09-29 20:16:33 on dist-test-slave-4cx3"
server_key: "e556814529080851aac1267c4c9b7157"
server_key_iv: "b4738734782d172e9a18fd62b98c2b48"
server_key_version: "encryptionkey@0"
I20260929 20:16:33.893929 30831 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:16:33.897367 30846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:33.898715 30831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:16:33.898893 30831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "458a6c4473b9465e9cdc1a7869524cd7"
format_stamp: "Formatted at 2026-09-29 20:16:33 on dist-test-slave-4cx3"
server_key: "e556814529080851aac1267c4c9b7157"
server_key_iv: "b4738734782d172e9a18fd62b98c2b48"
server_key_version: "encryptionkey@0"
I20260929 20:16:33.899102 30831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:33.926375 30831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:33.930787 30831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:33.931080 30831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:33.949517 30831 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:43481
I20260929 20:16:33.949539 30897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:43481 every 8 connection(s)
I20260929 20:16:33.950968 30831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:16:33.953081 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 30831
I20260929 20:16:33.953284 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:16:33.953919 28389 external_mini_cluster.cc:1428] Setting key cf7cab6f0322227b80eb0c5666b15b7d
I20260929 20:16:33.955107 30898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 20:16:33 dist-test-slave-4cx3 krb5kdc[30818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:33.969257 30898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 458a6c4473b9465e9cdc1a7869524cd7: Bootstrap starting.
I20260929 20:16:33.973557 30898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 458a6c4473b9465e9cdc1a7869524cd7: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:33.974898 30898 log.cc:824] T 00000000000000000000000000000000 P 458a6c4473b9465e9cdc1a7869524cd7: Log is configured to *not* fsync() on all Append() calls
I20260929 20:16:33.978329 30898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 458a6c4473b9465e9cdc1a7869524cd7: No bootstrap required, opened a new log
I20260929 20:16:33.978499 30901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:33.955950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36666 (local address 127.27.185.126:43481)
0929 20:16:33.956847 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:33.956862 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:33.956910 (+ 48us) server_negotiation.cc:407] Connection header received
0929 20:16:33.958737 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:33.958765 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:33.959296 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:33.959784 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:33.961059 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:33.963131 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:33.964437 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:33.965494 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:33.965692 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:33.969881 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:33.969892 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:33.969916 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:33.969971 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:33.973968 (+ 3997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:33.974667 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:33.974681 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:33.974692 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:33.974907 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:33.975297 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:33.975304 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:33.975310 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:33.976029 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:33.976298 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:33.976723 (+ 425us) server_negotiation.cc:299] Negotiation successful
0929 20:16:33.977147 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":365,"threads_started":1}
I20260929 20:16:33.986310 30898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 458a6c4473b9465e9cdc1a7869524cd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458a6c4473b9465e9cdc1a7869524cd7" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 43481 } }
I20260929 20:16:33.986701 30898 raft_consensus.cc:374] T 00000000000000000000000000000000 P 458a6c4473b9465e9cdc1a7869524cd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:33.986778 30898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 458a6c4473b9465e9cdc1a7869524cd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 458a6c4473b9465e9cdc1a7869524cd7, State: Initialized, Role: FOLLOWER
I20260929 20:16:33.987330 30898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 458a6c4473b9465e9cdc1a7869524cd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458a6c4473b9465e9cdc1a7869524cd7" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 43481 } }
I20260929 20:16:33.987596 30898 raft_consensus.cc:388] T 00000000000000000000000000000000 P 458a6c4473b9465e9cdc1a7869524cd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:16:33.987701 30898 raft_consensus.cc:482] T 00000000000000000000000000000000 P 458a6c4473b9465e9cdc1a7869524cd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:16:33.987820 30898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 458a6c4473b9465e9cdc1a7869524cd7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:33.989897 30898 raft_consensus.cc:504] T 00000000000000000000000000000000 P 458a6c4473b9465e9cdc1a7869524cd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458a6c4473b9465e9cdc1a7869524cd7" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 43481 } }
I20260929 20:16:33.990332 30898 leader_election.cc:302] T 00000000000000000000000000000000 P 458a6c4473b9465e9cdc1a7869524cd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 458a6c4473b9465e9cdc1a7869524cd7; no voters:
I20260929 20:16:33.990845 30898 leader_election.cc:288] T 00000000000000000000000000000000 P 458a6c4473b9465e9cdc1a7869524cd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:16:33.990960 30903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 458a6c4473b9465e9cdc1a7869524cd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:16:33.992340 30903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 458a6c4473b9465e9cdc1a7869524cd7 [term 1 LEADER]: Becoming Leader. State: Replica: 458a6c4473b9465e9cdc1a7869524cd7, State: Running, Role: LEADER
I20260929 20:16:33.992901 30903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 458a6c4473b9465e9cdc1a7869524cd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458a6c4473b9465e9cdc1a7869524cd7" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 43481 } }
I20260929 20:16:33.993212 30898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 458a6c4473b9465e9cdc1a7869524cd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:16:33.996371 30904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 458a6c4473b9465e9cdc1a7869524cd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "458a6c4473b9465e9cdc1a7869524cd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458a6c4473b9465e9cdc1a7869524cd7" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 43481 } } }
I20260929 20:16:33.996783 30904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 458a6c4473b9465e9cdc1a7869524cd7 [sys.catalog]: This master's current role is: LEADER
I20260929 20:16:33.996343 30905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 458a6c4473b9465e9cdc1a7869524cd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 458a6c4473b9465e9cdc1a7869524cd7. Latest consensus state: current_term: 1 leader_uuid: "458a6c4473b9465e9cdc1a7869524cd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458a6c4473b9465e9cdc1a7869524cd7" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 43481 } } }
I20260929 20:16:33.997129 30905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 458a6c4473b9465e9cdc1a7869524cd7 [sys.catalog]: This master's current role is: LEADER
I20260929 20:16:33.998616 30911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:16:34.005149 30911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:16:34.014969 30911 catalog_manager.cc:1347] Generated new cluster ID: 5309d5cf68dc48abbae5d1d243063584
I20260929 20:16:34.015100 30911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:16:34.038645 30911 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:16:34.040117 30911 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:16:34.047328 30911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 458a6c4473b9465e9cdc1a7869524cd7: Generated new TSK 0
I20260929 20:16:34.048506 30911 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
I20260929 20:16:34.109059 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:0
--local_ip_for_outbound_sockets=127.27.185.65
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:43481
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:33805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:34.441365 30924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:34.441656 30924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:34.441815 30924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:34.449229 30924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:34.449354 30924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:34.449525 30924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:16:34.459355 30924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:33805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:43481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:34.460672 30924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:34.462019 30924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:34.475188 30930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:34.475143 30932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:34.475284 30929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:34.476125 30924 server_base.cc:1034] running on GCE node
I20260929 20:16:34.476737 30924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:34.477945 30924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:34.479187 30924 hybrid_clock.cc:648] HybridClock initialized: now 1790712994479127 us; error 50 us; skew 500 ppm
I20260929 20:16:34.479569 30924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:34 dist-test-slave-4cx3 krb5kdc[30818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712994, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:34.482837 30924 init.cc:377] Logged in from keytab as kudu/127.27.185.65@KRBTEST.COM (short username kudu)
I20260929 20:16:34.484504 30924 webserver.cc:460] Webserver started at http://127.27.185.65:39105/ using document root <none> and password file <none>
I20260929 20:16:34.485211 30924 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:34.485354 30924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:34.485690 30924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:34.488818 30924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/data/instance:
uuid: "12c3f1677c0c4f9b98c65c5b13165db2"
format_stamp: "Formatted at 2026-09-29 20:16:34 on dist-test-slave-4cx3"
server_key: "0dbdf712292921721fe9048cd023d7cd"
server_key_iv: "03edebf0eefe0ebca73bc97d5661f957"
server_key_version: "encryptionkey@0"
I20260929 20:16:34.489560 30924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance:
uuid: "12c3f1677c0c4f9b98c65c5b13165db2"
format_stamp: "Formatted at 2026-09-29 20:16:34 on dist-test-slave-4cx3"
server_key: "0dbdf712292921721fe9048cd023d7cd"
server_key_iv: "03edebf0eefe0ebca73bc97d5661f957"
server_key_version: "encryptionkey@0"
I20260929 20:16:34.495271 30924 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:16:34.498916 30939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:34.500380 30924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:16:34.500581 30924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "12c3f1677c0c4f9b98c65c5b13165db2"
format_stamp: "Formatted at 2026-09-29 20:16:34 on dist-test-slave-4cx3"
server_key: "0dbdf712292921721fe9048cd023d7cd"
server_key_iv: "03edebf0eefe0ebca73bc97d5661f957"
server_key_version: "encryptionkey@0"
I20260929 20:16:34.500770 30924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:34.522188 30924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:34.526683 30924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:34.526927 30924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:34.527951 30924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:16:34.529330 30924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:16:34.529409 30924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:34.529505 30924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:16:34.529552 30924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:34.563061 30924 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:45305
I20260929 20:16:34.563110 31053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:45305 every 8 connection(s)
I20260929 20:16:34.564518 30924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:16:34.570459 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 30924
I20260929 20:16:34.570664 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance
I20260929 20:16:34.571503 28389 external_mini_cluster.cc:1428] Setting key 2797dd3803030b5835c32ea6fa09fde7
Sep 29 20:16:34 dist-test-slave-4cx3 krb5kdc[30818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712994, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:34.589433 31057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:34.567307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:46459 (local address 127.27.185.126:43481)
0929 20:16:34.567864 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:34.567869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:34.568773 (+ 904us) server_negotiation.cc:407] Connection header received
0929 20:16:34.571475 (+ 2702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:34.571482 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:34.571595 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:34.571738 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:34.573592 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:34.574694 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:34.576391 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:34.577223 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:34.577393 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:34.581285 (+ 3892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:34.581295 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:34.581303 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:34.581353 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:34.585275 (+ 3922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:34.586474 (+ 1199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:34.586484 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:34.586490 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:34.586656 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:34.587146 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:34.587155 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:34.587161 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:34.587593 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:34.587760 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:34.588605 (+ 845us) server_negotiation.cc:299] Negotiation successful
0929 20:16:34.588903 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":320,"threads_started":1}
I20260929 20:16:34.590178 31058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:34.567882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:43481 (local address 127.27.185.65:46459)
0929 20:16:34.568612 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:34.568652 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:34.571020 (+ 2368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:34.572043 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:34.572056 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:34.572795 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:34.573408 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:34.573426 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:34.574859 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:34.574863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:34.576232 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:34.576246 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:34.577481 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:34.577487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:34.577711 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:34.578422 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:34.578457 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:34.581005 (+ 2548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:34.585474 (+ 4469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:34.585482 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:34.585497 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:34.586272 (+ 775us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:34.586786 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:34.586792 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:34.586795 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:34.587010 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:34.587721 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:34.587729 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:34.588179 (+ 450us) client_negotiation.cc:769] Sending connection context
0929 20:16:34.588435 (+ 256us) client_negotiation.cc:240] Negotiation successful
0929 20:16:34.588960 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":267,"threads_started":1}
I20260929 20:16:34.591903 31054 heartbeater.cc:347] Connected to a master server at 127.27.185.126:43481
WARNING: no policy specified for kudu/127.27.185.66@KRBTEST.COM; defaulting to no policy
I20260929 20:16:34.592267 31054 heartbeater.cc:464] Registering TS with master...
I20260929 20:16:34.593070 31054 heartbeater.cc:511] Master 127.27.185.126:43481 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.66@KRBTEST.COM" created.
I20260929 20:16:34.595953 30863 ts_manager.cc:194] Registered new tserver with Master: 12c3f1677c0c4f9b98c65c5b13165db2 (127.27.185.65:45305)
I20260929 20:16:34.597575 30863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.65@KRBTEST.COM'} at 127.27.185.65:46459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
I20260929 20:16:34.619019 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:0
--local_ip_for_outbound_sockets=127.27.185.66
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:43481
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:33805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:34.942104 31061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:34.942375 31061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:34.942552 31061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:34.949788 31061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:34.949887 31061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:34.950083 31061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:16:34.960019 31061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:33805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:43481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:34.961467 31061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:34.962785 31061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:34.976142 31066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:34.976262 31067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:34.976163 31069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:34.976682 31061 server_base.cc:1034] running on GCE node
I20260929 20:16:34.977403 31061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:34.978351 31061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:34.979570 31061 hybrid_clock.cc:648] HybridClock initialized: now 1790712994979527 us; error 55 us; skew 500 ppm
I20260929 20:16:34.979892 31061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:34 dist-test-slave-4cx3 krb5kdc[30818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712994, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:34.987291 31061 init.cc:377] Logged in from keytab as kudu/127.27.185.66@KRBTEST.COM (short username kudu)
I20260929 20:16:34.988799 31061 webserver.cc:460] Webserver started at http://127.27.185.66:41161/ using document root <none> and password file <none>
I20260929 20:16:34.989562 31061 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:34.989665 31061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:34.989928 31061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:34.992915 31061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/data/instance:
uuid: "dc9e390468b141f5afde250fe9ae1fd6"
format_stamp: "Formatted at 2026-09-29 20:16:34 on dist-test-slave-4cx3"
server_key: "4ee8ff7f6dc4ad4bb461634c256e7031"
server_key_iv: "50881ee80c657b5b40600aa9eca86357"
server_key_version: "encryptionkey@0"
I20260929 20:16:34.993695 31061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance:
uuid: "dc9e390468b141f5afde250fe9ae1fd6"
format_stamp: "Formatted at 2026-09-29 20:16:34 on dist-test-slave-4cx3"
server_key: "4ee8ff7f6dc4ad4bb461634c256e7031"
server_key_iv: "50881ee80c657b5b40600aa9eca86357"
server_key_version: "encryptionkey@0"
I20260929 20:16:34.998823 31061 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:16:35.002399 31076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:35.003863 31061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:16:35.004032 31061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "dc9e390468b141f5afde250fe9ae1fd6"
format_stamp: "Formatted at 2026-09-29 20:16:34 on dist-test-slave-4cx3"
server_key: "4ee8ff7f6dc4ad4bb461634c256e7031"
server_key_iv: "50881ee80c657b5b40600aa9eca86357"
server_key_version: "encryptionkey@0"
I20260929 20:16:35.004279 31061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:35.026525 31061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:35.030648 31061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:35.030951 31061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:35.031978 31061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:16:35.033390 31061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:16:35.033501 31061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:35.033648 31061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:16:35.033716 31061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:35.066797 31061 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:42883
I20260929 20:16:35.066859 31189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:42883 every 8 connection(s)
I20260929 20:16:35.068197 31061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:16:35.077761 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 31061
I20260929 20:16:35.077951 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance
I20260929 20:16:35.078394 28389 external_mini_cluster.cc:1428] Setting key 64c2d55547ee87619e4b49660f445a1b
Sep 29 20:16:35 dist-test-slave-4cx3 krb5kdc[30818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712994, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:35.089537 31057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:35.070796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:43761 (local address 127.27.185.126:43481)
0929 20:16:35.070968 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:35.070972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:35.072229 (+ 1257us) server_negotiation.cc:407] Connection header received
0929 20:16:35.074065 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:35.074071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:35.074165 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:35.074275 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:35.075835 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:35.076559 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:35.078180 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:35.078948 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:35.079095 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:35.083005 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:35.083010 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:35.083013 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:35.083046 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:35.086152 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:35.086967 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:35.086972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:35.086974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:35.087158 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:35.087686 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:35.087692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:35.087695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:35.087985 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:35.088141 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:35.088907 (+ 766us) server_negotiation.cc:299] Negotiation successful
0929 20:16:35.089159 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:16:35.090698 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:35.071324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:43481 (local address 127.27.185.66:43761)
0929 20:16:35.072073 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:35.072115 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:35.073785 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:35.074526 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:35.074534 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:35.075044 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:35.075666 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:35.075684 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:35.076718 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:35.076721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:35.078041 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:35.078056 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:35.079204 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:35.079211 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:35.079581 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:35.080298 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:35.080320 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:35.082759 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:35.086318 (+ 3559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:35.086329 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:35.086349 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:35.086819 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:35.087272 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:35.087278 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:35.087281 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:35.087566 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:35.088123 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:35.088132 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:35.088525 (+ 393us) client_negotiation.cc:769] Sending connection context
0929 20:16:35.088798 (+ 273us) client_negotiation.cc:240] Negotiation successful
0929 20:16:35.089370 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":265,"threads_started":1}
I20260929 20:16:35.092420 31190 heartbeater.cc:347] Connected to a master server at 127.27.185.126:43481
I20260929 20:16:35.092743 31190 heartbeater.cc:464] Registering TS with master...
I20260929 20:16:35.093569 31190 heartbeater.cc:511] Master 127.27.185.126:43481 requested a full tablet report, sending...
I20260929 20:16:35.095149 30863 ts_manager.cc:194] Registered new tserver with Master: dc9e390468b141f5afde250fe9ae1fd6 (127.27.185.66:42883)
I20260929 20:16:35.096009 30863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.66@KRBTEST.COM'} at 127.27.185.66:43761
WARNING: no policy specified for kudu/127.27.185.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
I20260929 20:16:35.123169 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:0
--local_ip_for_outbound_sockets=127.27.185.67
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:43481
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:33805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:35.445499 31195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:35.445801 31195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:35.445895 31195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:35.453111 31195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:35.453225 31195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:35.453346 31195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:16:35.462939 31195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:33805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:43481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:35.464195 31195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:35.465464 31195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:35.478657 31201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:35.478711 31203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:35.478798 31200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:35.479362 31195 server_base.cc:1034] running on GCE node
I20260929 20:16:35.480065 31195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:35.480970 31195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:35.482170 31195 hybrid_clock.cc:648] HybridClock initialized: now 1790712995482117 us; error 47 us; skew 500 ppm
I20260929 20:16:35.482448 31195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:35 dist-test-slave-4cx3 krb5kdc[30818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712995, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:35.485551 31195 init.cc:377] Logged in from keytab as kudu/127.27.185.67@KRBTEST.COM (short username kudu)
I20260929 20:16:35.487092 31195 webserver.cc:460] Webserver started at http://127.27.185.67:39693/ using document root <none> and password file <none>
I20260929 20:16:35.487872 31195 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:35.488003 31195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:35.488274 31195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:35.491195 31195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/data/instance:
uuid: "e504a96622ca4491b4759519bd211599"
format_stamp: "Formatted at 2026-09-29 20:16:35 on dist-test-slave-4cx3"
server_key: "7c8ac2bbfc589323d77bbe827d031fca"
server_key_iv: "bcf6f186015f1a54b4cf897a7652eb87"
server_key_version: "encryptionkey@0"
I20260929 20:16:35.491993 31195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance:
uuid: "e504a96622ca4491b4759519bd211599"
format_stamp: "Formatted at 2026-09-29 20:16:35 on dist-test-slave-4cx3"
server_key: "7c8ac2bbfc589323d77bbe827d031fca"
server_key_iv: "bcf6f186015f1a54b4cf897a7652eb87"
server_key_version: "encryptionkey@0"
I20260929 20:16:35.497246 31195 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:16:35.500784 31210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:35.502229 31195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:16:35.502421 31195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "e504a96622ca4491b4759519bd211599"
format_stamp: "Formatted at 2026-09-29 20:16:35 on dist-test-slave-4cx3"
server_key: "7c8ac2bbfc589323d77bbe827d031fca"
server_key_iv: "bcf6f186015f1a54b4cf897a7652eb87"
server_key_version: "encryptionkey@0"
I20260929 20:16:35.502653 31195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:35.519088 31195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:35.523609 31195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:35.523857 31195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:35.524785 31195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:16:35.526068 31195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:16:35.526144 31195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:35.526237 31195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:16:35.526289 31195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:35.559495 31195 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:41709
I20260929 20:16:35.559505 31323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:41709 every 8 connection(s)
I20260929 20:16:35.560817 31195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:16:35.561213 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 31195
I20260929 20:16:35.561383 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance
I20260929 20:16:35.562013 28389 external_mini_cluster.cc:1428] Setting key 56a0e891d672b909fd5194a8572935e0
Sep 29 20:16:35 dist-test-slave-4cx3 krb5kdc[30818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712995, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:35.582154 31057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:35.563721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:52933 (local address 127.27.185.126:43481)
0929 20:16:35.563973 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:35.563980 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:35.565568 (+ 1588us) server_negotiation.cc:407] Connection header received
0929 20:16:35.567472 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:35.567478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:35.567563 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:35.567677 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:35.569175 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:35.569917 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:35.571449 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:35.571870 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:35.572011 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:35.576276 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:35.576283 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:35.576288 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:35.576332 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:35.579129 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:35.579918 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:35.579921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:35.579924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:35.580055 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:35.580654 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:35.580657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:35.580659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:35.580918 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:35.581043 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:35.581732 (+ 689us) server_negotiation.cc:299] Negotiation successful
0929 20:16:35.581890 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260929 20:16:35.583251 31326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:35.564485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:43481 (local address 127.27.185.67:52933)
0929 20:16:35.565411 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:35.565451 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:35.567159 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:35.567909 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:35.567918 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:35.568424 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:35.569007 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:35.569023 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:35.570087 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:35.570092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:35.571290 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:35.571301 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:35.572096 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:35.572106 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:35.572340 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:35.572964 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:35.572989 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:35.576006 (+ 3017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:35.579271 (+ 3265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:35.579278 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:35.579291 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:35.579812 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:35.580285 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:35.580339 (+ 54us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:35.580342 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:35.580552 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:35.581054 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:35.581060 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:35.581396 (+ 336us) client_negotiation.cc:769] Sending connection context
0929 20:16:35.581659 (+ 263us) client_negotiation.cc:240] Negotiation successful
0929 20:16:35.582052 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":372,"threads_started":1}
I20260929 20:16:35.584719 31324 heartbeater.cc:347] Connected to a master server at 127.27.185.126:43481
I20260929 20:16:35.585041 31324 heartbeater.cc:464] Registering TS with master...
I20260929 20:16:35.585764 31324 heartbeater.cc:511] Master 127.27.185.126:43481 requested a full tablet report, sending...
I20260929 20:16:35.587160 30863 ts_manager.cc:194] Registered new tserver with Master: e504a96622ca4491b4759519bd211599 (127.27.185.67:41709)
I20260929 20:16:35.588079 30863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.67@KRBTEST.COM'} at 127.27.185.67:52933
I20260929 20:16:35.590380 28389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:16:35.601001 31054 heartbeater.cc:503] Master 127.27.185.126:43481 was elected leader, sending a full tablet report...
W20260929 20:16:35.620929 31057 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:16:35.618547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36682 (local address 127.27.185.126:43481)
0929 20:16:35.618714 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:35.618719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:35.619219 (+ 500us) server_negotiation.cc:407] Connection header received
0929 20:16:35.619367 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:35.619371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:35.619475 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:35.619574 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:35.620745 (+ 1171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36682: BlockingRecv error: recv got EOF from 127.0.0.1:36682 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:16:35.624063 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 30924
W20260929 20:16:35.626739 28389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 30924: Not authorized: Client connection negotiation failed: client connection to 127.27.185.65:45305: server requires authentication, but client does not have Kerberos credentials available
I20260929 20:16:35.626937 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 30924
I20260929 20:16:35.642270 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 31061
W20260929 20:16:35.644805 28389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 31061: Not authorized: Client connection negotiation failed: client connection to 127.27.185.66:42883: server requires authentication, but client does not have Kerberos credentials available
I20260929 20:16:35.644917 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 31061
I20260929 20:16:35.660386 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 31195
W20260929 20:16:35.663115 28389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 31195: Not authorized: Client connection negotiation failed: client connection to 127.27.185.67:41709: server requires authentication, but client does not have Kerberos credentials available
I20260929 20:16:35.663300 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 31195
I20260929 20:16:35.678546 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 30831
I20260929 20:16:35.679735 30893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:16:35.781836 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 30831
2026-09-29T20:16:35Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3555 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:16:35 dist-test-slave-4cx3 krb5kdc[31341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:16:35 dist-test-slave-4cx3 krb5kdc[31341](info): set up 2 sockets
Sep 29 20:16:35 dist-test-slave-4cx3 krb5kdc[31341](info): commencing operation
krb5kdc: starting...
W20260929 20:16:36.878794 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:16:37 dist-test-slave-4cx3 krb5kdc[31341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:16:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:16:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
I20260929 20:16:37.075692 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:39007
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:37615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:39007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:37.397526 31354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:37.397945 31354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:37.398135 31354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:37.408090 31354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:16:37.408195 31354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:37.408270 31354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:37.408362 31354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:16:37.408427 31354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:16:37.418345 31354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:37615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:39007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:39007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:37.419723 31354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:37.421016 31354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:37.433101 31359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:37.433059 31360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:37.433000 31362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:37.433836 31354 server_base.cc:1034] running on GCE node
I20260929 20:16:37.434636 31354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:37.436137 31354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:37.437395 31354 hybrid_clock.cc:648] HybridClock initialized: now 1790712997437359 us; error 44 us; skew 500 ppm
I20260929 20:16:37.437741 31354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:37 dist-test-slave-4cx3 krb5kdc[31341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712997, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:37.440939 31354 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:16:37.442391 31354 webserver.cc:460] Webserver started at http://127.27.185.126:34363/ using document root <none> and password file <none>
I20260929 20:16:37.443023 31354 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:37.443121 31354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:37.443430 31354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:37.446835 31354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "9f782632766c4a1e8c0f67fbbe11cfb3"
format_stamp: "Formatted at 2026-09-29 20:16:37 on dist-test-slave-4cx3"
server_key: "da88a41b6c7a2a2a14262dcee45ba6b1"
server_key_iv: "941d1b8f6aa88c0b9f0292a5021fa98e"
server_key_version: "encryptionkey@0"
I20260929 20:16:37.447666 31354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "9f782632766c4a1e8c0f67fbbe11cfb3"
format_stamp: "Formatted at 2026-09-29 20:16:37 on dist-test-slave-4cx3"
server_key: "da88a41b6c7a2a2a14262dcee45ba6b1"
server_key_iv: "941d1b8f6aa88c0b9f0292a5021fa98e"
server_key_version: "encryptionkey@0"
I20260929 20:16:37.452615 31354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:16:37.456017 31369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:37.457293 31354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 20:16:37.457479 31354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "9f782632766c4a1e8c0f67fbbe11cfb3"
format_stamp: "Formatted at 2026-09-29 20:16:37 on dist-test-slave-4cx3"
server_key: "da88a41b6c7a2a2a14262dcee45ba6b1"
server_key_iv: "941d1b8f6aa88c0b9f0292a5021fa98e"
server_key_version: "encryptionkey@0"
I20260929 20:16:37.457681 31354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:37.506206 31354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:37.511782 31354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:37.512079 31354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:37.535653 31354 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:39007
I20260929 20:16:37.535672 31421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:39007 every 8 connection(s)
I20260929 20:16:37.537163 31354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:16:37.541555 31422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:37.544643 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 31354
I20260929 20:16:37.544914 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:16:37.545783 28389 external_mini_cluster.cc:1428] Setting key f0a28e31465000003e0c07e4ce718c9b
I20260929 20:16:37.555365 31422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f782632766c4a1e8c0f67fbbe11cfb3: Bootstrap starting.
I20260929 20:16:37.558964 31422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f782632766c4a1e8c0f67fbbe11cfb3: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:37.560045 31422 log.cc:824] T 00000000000000000000000000000000 P 9f782632766c4a1e8c0f67fbbe11cfb3: Log is configured to *not* fsync() on all Append() calls
Sep 29 20:16:37 dist-test-slave-4cx3 krb5kdc[31341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:37.563113 31422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f782632766c4a1e8c0f67fbbe11cfb3: No bootstrap required, opened a new log
I20260929 20:16:37.570161 31425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:37.547778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41756 (local address 127.27.185.126:39007)
0929 20:16:37.548597 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:37.548613 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:37.548661 (+ 48us) server_negotiation.cc:407] Connection header received
0929 20:16:37.551076 (+ 2415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:37.551109 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:37.551685 (+ 576us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:37.552156 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:37.553509 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:37.555554 (+ 2045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:37.556795 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:37.557900 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:37.558084 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:37.562367 (+ 4283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:37.562375 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:37.562389 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:37.562422 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:37.565640 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:37.566518 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:37.566525 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:37.566532 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:37.566706 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:37.567127 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:37.567131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:37.567133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:37.567743 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:37.567996 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:37.568529 (+ 533us) server_negotiation.cc:299] Negotiation successful
0929 20:16:37.568961 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":338,"threads_started":1}
I20260929 20:16:37.572491 31422 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f782632766c4a1e8c0f67fbbe11cfb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f782632766c4a1e8c0f67fbbe11cfb3" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 39007 } }
I20260929 20:16:37.572983 31422 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f782632766c4a1e8c0f67fbbe11cfb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:37.573105 31422 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f782632766c4a1e8c0f67fbbe11cfb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f782632766c4a1e8c0f67fbbe11cfb3, State: Initialized, Role: FOLLOWER
I20260929 20:16:37.573796 31422 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f782632766c4a1e8c0f67fbbe11cfb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f782632766c4a1e8c0f67fbbe11cfb3" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 39007 } }
I20260929 20:16:37.574090 31422 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f782632766c4a1e8c0f67fbbe11cfb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:16:37.574219 31422 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f782632766c4a1e8c0f67fbbe11cfb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:16:37.574424 31422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f782632766c4a1e8c0f67fbbe11cfb3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:37.574658 28389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:16:37.576725 31422 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f782632766c4a1e8c0f67fbbe11cfb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f782632766c4a1e8c0f67fbbe11cfb3" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 39007 } }
I20260929 20:16:37.577284 31422 leader_election.cc:302] T 00000000000000000000000000000000 P 9f782632766c4a1e8c0f67fbbe11cfb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f782632766c4a1e8c0f67fbbe11cfb3; no voters:
I20260929 20:16:37.577955 31422 leader_election.cc:288] T 00000000000000000000000000000000 P 9f782632766c4a1e8c0f67fbbe11cfb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:16:37.578095 31427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f782632766c4a1e8c0f67fbbe11cfb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:16:37.579452 31427 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f782632766c4a1e8c0f67fbbe11cfb3 [term 1 LEADER]: Becoming Leader. State: Replica: 9f782632766c4a1e8c0f67fbbe11cfb3, State: Running, Role: LEADER
I20260929 20:16:37.580127 31427 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f782632766c4a1e8c0f67fbbe11cfb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f782632766c4a1e8c0f67fbbe11cfb3" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 39007 } }
I20260929 20:16:37.580574 31422 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f782632766c4a1e8c0f67fbbe11cfb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:16:37.585100 31429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f782632766c4a1e8c0f67fbbe11cfb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f782632766c4a1e8c0f67fbbe11cfb3. Latest consensus state: current_term: 1 leader_uuid: "9f782632766c4a1e8c0f67fbbe11cfb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f782632766c4a1e8c0f67fbbe11cfb3" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 39007 } } }
I20260929 20:16:37.585510 31429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f782632766c4a1e8c0f67fbbe11cfb3 [sys.catalog]: This master's current role is: LEADER
I20260929 20:16:37.585064 31428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f782632766c4a1e8c0f67fbbe11cfb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f782632766c4a1e8c0f67fbbe11cfb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f782632766c4a1e8c0f67fbbe11cfb3" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 39007 } } }
I20260929 20:16:37.586126 31428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f782632766c4a1e8c0f67fbbe11cfb3 [sys.catalog]: This master's current role is: LEADER
I20260929 20:16:37.593235 31434 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:16:37.599524 31434 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:16:37.620857 31434 catalog_manager.cc:1347] Generated new cluster ID: f978a26745324cca8a7908fc46c3370b
I20260929 20:16:37.621101 31434 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:16:37.644210 31425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:37.634858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41762 (local address 127.27.185.126:39007)
0929 20:16:37.635027 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:37.635033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:37.635968 (+ 935us) server_negotiation.cc:407] Connection header received
0929 20:16:37.636136 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:37.636142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:37.636233 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:37.636343 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:37.637507 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:37.638727 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:37.640541 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:37.641219 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:37.641350 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:37.642486 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:37.642497 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:37.642502 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:16:37.643359 (+ 857us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:37.643463 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:16:37.643500 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:37.643609 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:37.643730 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 20:16:37.643824 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":58}
I20260929 20:16:37.655236 31434 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:16:37.656648 31434 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:16:37.667115 31434 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f782632766c4a1e8c0f67fbbe11cfb3: Generated new TSK 0
I20260929 20:16:37.668033 31434 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:16:37.678947 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 31354
I20260929 20:16:37.680181 31417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:16:37.772792 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 31354
2026-09-29T20:16:37Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1985 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:16:37 dist-test-slave-4cx3 krb5kdc[31456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:16:37 dist-test-slave-4cx3 krb5kdc[31456](info): set up 2 sockets
Sep 29 20:16:37 dist-test-slave-4cx3 krb5kdc[31456](info): commencing operation
krb5kdc: starting...
W20260929 20:16:38.927032 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:16:39 dist-test-slave-4cx3 krb5kdc[31456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:16:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:16:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
I20260929 20:16:39.124413 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:41617
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:34791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:41617
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:39.449554 31469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:39.449869 31469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:39.449960 31469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:39.457314 31469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:16:39.457439 31469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:39.457485 31469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:39.457542 31469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:16:39.457587 31469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:16:39.467679 31469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:34791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:41617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:41617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:39.469025 31469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:39.470343 31469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:39.482249 31475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:39.482209 31477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:39.482465 31474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:39.482801 31469 server_base.cc:1034] running on GCE node
I20260929 20:16:39.483718 31469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:39.485186 31469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:39.486428 31469 hybrid_clock.cc:648] HybridClock initialized: now 1790712999486342 us; error 73 us; skew 500 ppm
I20260929 20:16:39.486783 31469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:39 dist-test-slave-4cx3 krb5kdc[31456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712999, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:39.489928 31469 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:16:39.491609 31469 webserver.cc:460] Webserver started at http://127.27.185.126:44495/ using document root <none> and password file <none>
I20260929 20:16:39.492339 31469 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:39.492489 31469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:39.492807 31469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:39.495985 31469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "9d956bd9cb384df6b542a235020c326a"
format_stamp: "Formatted at 2026-09-29 20:16:39 on dist-test-slave-4cx3"
server_key: "55adf502842634d5c32f90e730be6cfa"
server_key_iv: "8afcae69c47c1af1a264ed5e2e49d46a"
server_key_version: "encryptionkey@0"
I20260929 20:16:39.496783 31469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "9d956bd9cb384df6b542a235020c326a"
format_stamp: "Formatted at 2026-09-29 20:16:39 on dist-test-slave-4cx3"
server_key: "55adf502842634d5c32f90e730be6cfa"
server_key_iv: "8afcae69c47c1af1a264ed5e2e49d46a"
server_key_version: "encryptionkey@0"
I20260929 20:16:39.502452 31469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 20:16:39.506120 31484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:39.507622 31469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:16:39.507833 31469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "9d956bd9cb384df6b542a235020c326a"
format_stamp: "Formatted at 2026-09-29 20:16:39 on dist-test-slave-4cx3"
server_key: "55adf502842634d5c32f90e730be6cfa"
server_key_iv: "8afcae69c47c1af1a264ed5e2e49d46a"
server_key_version: "encryptionkey@0"
I20260929 20:16:39.508042 31469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:39.523476 31469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:39.527639 31469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:39.527882 31469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:39.545091 31469 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:41617
I20260929 20:16:39.545118 31535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:41617 every 8 connection(s)
I20260929 20:16:39.546504 31469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:16:39.550379 31536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:39.552757 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 31469
I20260929 20:16:39.552984 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:16:39.553584 28389 external_mini_cluster.cc:1428] Setting key 7f87df28ae0c1effe905bacd1a9446d0
I20260929 20:16:39.562690 31536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d956bd9cb384df6b542a235020c326a: Bootstrap starting.
I20260929 20:16:39.566744 31536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d956bd9cb384df6b542a235020c326a: Neither blocks nor log segments found. Creating new log.
Sep 29 20:16:39 dist-test-slave-4cx3 krb5kdc[31456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790712999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:39.568069 31536 log.cc:824] T 00000000000000000000000000000000 P 9d956bd9cb384df6b542a235020c326a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:16:39.571437 31536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d956bd9cb384df6b542a235020c326a: No bootstrap required, opened a new log
I20260929 20:16:39.576645 31539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:39.555551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41160 (local address 127.27.185.126:41617)
0929 20:16:39.556362 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:39.556374 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:39.556414 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:16:39.558032 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:39.558054 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:39.558426 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:39.558746 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:39.560516 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:39.561785 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:39.563527 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:39.564211 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:39.564355 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:39.569174 (+ 4819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:39.569183 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:39.569202 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:39.569248 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:39.572404 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:39.573150 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:39.573160 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:39.573166 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:39.573334 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:39.573756 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:39.573769 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:39.573771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:39.574340 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:39.574688 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:39.575108 (+ 420us) server_negotiation.cc:299] Negotiation successful
0929 20:16:39.575589 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":374,"threads_started":1}
I20260929 20:16:39.580014 31536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d956bd9cb384df6b542a235020c326a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d956bd9cb384df6b542a235020c326a" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41617 } }
I20260929 20:16:39.580425 31536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d956bd9cb384df6b542a235020c326a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:39.580533 31536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d956bd9cb384df6b542a235020c326a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d956bd9cb384df6b542a235020c326a, State: Initialized, Role: FOLLOWER
I20260929 20:16:39.581065 31536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d956bd9cb384df6b542a235020c326a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d956bd9cb384df6b542a235020c326a" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41617 } }
I20260929 20:16:39.581287 31536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d956bd9cb384df6b542a235020c326a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:16:39.581439 31536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d956bd9cb384df6b542a235020c326a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:16:39.581610 31536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d956bd9cb384df6b542a235020c326a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:39.584285 31536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d956bd9cb384df6b542a235020c326a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d956bd9cb384df6b542a235020c326a" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41617 } }
I20260929 20:16:39.584841 31536 leader_election.cc:302] T 00000000000000000000000000000000 P 9d956bd9cb384df6b542a235020c326a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d956bd9cb384df6b542a235020c326a; no voters:
I20260929 20:16:39.585407 31536 leader_election.cc:288] T 00000000000000000000000000000000 P 9d956bd9cb384df6b542a235020c326a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:16:39.585546 31541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d956bd9cb384df6b542a235020c326a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:16:39.586249 31541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d956bd9cb384df6b542a235020c326a [term 1 LEADER]: Becoming Leader. State: Replica: 9d956bd9cb384df6b542a235020c326a, State: Running, Role: LEADER
I20260929 20:16:39.586844 31541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d956bd9cb384df6b542a235020c326a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d956bd9cb384df6b542a235020c326a" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41617 } }
I20260929 20:16:39.587673 31536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d956bd9cb384df6b542a235020c326a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:16:39.592983 31543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d956bd9cb384df6b542a235020c326a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d956bd9cb384df6b542a235020c326a. Latest consensus state: current_term: 1 leader_uuid: "9d956bd9cb384df6b542a235020c326a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d956bd9cb384df6b542a235020c326a" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41617 } } }
I20260929 20:16:39.593375 31543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d956bd9cb384df6b542a235020c326a [sys.catalog]: This master's current role is: LEADER
I20260929 20:16:39.593513 31542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d956bd9cb384df6b542a235020c326a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d956bd9cb384df6b542a235020c326a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d956bd9cb384df6b542a235020c326a" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41617 } } }
I20260929 20:16:39.593811 31542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d956bd9cb384df6b542a235020c326a [sys.catalog]: This master's current role is: LEADER
I20260929 20:16:39.595034 31548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:16:39.600515 31548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:16:39.609444 31548 catalog_manager.cc:1347] Generated new cluster ID: 9fb607b38a3f432786385dd278181eeb
I20260929 20:16:39.609575 31548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:16:39.624498 31548 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:16:39.626415 31548 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:16:39.633036 31548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d956bd9cb384df6b542a235020c326a: Generated new TSK 0
I20260929 20:16:39.633872 31548 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
I20260929 20:16:39.686632 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:0
--local_ip_for_outbound_sockets=127.27.185.65
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:41617
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:34791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:40.014968 31563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:40.015255 31563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:40.015349 31563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:40.022617 31563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:40.022735 31563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:40.022850 31563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:16:40.032660 31563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:34791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:41617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:40.033954 31563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:40.035239 31563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:40.048420 31568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:40.048396 31571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:40.048348 31569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:40.049249 31563 server_base.cc:1034] running on GCE node
I20260929 20:16:40.049938 31563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:40.050957 31563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:40.052186 31563 hybrid_clock.cc:648] HybridClock initialized: now 1790713000052132 us; error 47 us; skew 500 ppm
I20260929 20:16:40.052536 31563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:40 dist-test-slave-4cx3 krb5kdc[31456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713000, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:40.063182 31563 init.cc:377] Logged in from keytab as kudu/127.27.185.65@KRBTEST.COM (short username kudu)
I20260929 20:16:40.064877 31563 webserver.cc:460] Webserver started at http://127.27.185.65:38077/ using document root <none> and password file <none>
I20260929 20:16:40.065617 31563 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:40.065728 31563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:40.066009 31563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:40.069105 31563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/data/instance:
uuid: "ddd0906b46cc41fd86d22a922035b461"
format_stamp: "Formatted at 2026-09-29 20:16:40 on dist-test-slave-4cx3"
server_key: "dac522466f1cb55cf95a2df89fbefe5e"
server_key_iv: "edbf33affe43c480f0cc5a6ff3c4850f"
server_key_version: "encryptionkey@0"
I20260929 20:16:40.069950 31563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance:
uuid: "ddd0906b46cc41fd86d22a922035b461"
format_stamp: "Formatted at 2026-09-29 20:16:40 on dist-test-slave-4cx3"
server_key: "dac522466f1cb55cf95a2df89fbefe5e"
server_key_iv: "edbf33affe43c480f0cc5a6ff3c4850f"
server_key_version: "encryptionkey@0"
I20260929 20:16:40.076037 31563 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 20:16:40.079902 31578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:40.081444 31563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:16:40.081633 31563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "ddd0906b46cc41fd86d22a922035b461"
format_stamp: "Formatted at 2026-09-29 20:16:40 on dist-test-slave-4cx3"
server_key: "dac522466f1cb55cf95a2df89fbefe5e"
server_key_iv: "edbf33affe43c480f0cc5a6ff3c4850f"
server_key_version: "encryptionkey@0"
I20260929 20:16:40.081846 31563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:40.099964 31563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:40.104063 31563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:40.104349 31563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:40.105316 31563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:16:40.106632 31563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:16:40.106746 31563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:40.106884 31563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:16:40.106959 31563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:40.138643 31563 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:41393
I20260929 20:16:40.138679 31690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:41393 every 8 connection(s)
I20260929 20:16:40.140108 31563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:16:40.146181 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 31563
I20260929 20:16:40.146425 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance
I20260929 20:16:40.147254 28389 external_mini_cluster.cc:1428] Setting key f0ef086c45369f76d37007d2b594d474
Sep 29 20:16:40 dist-test-slave-4cx3 krb5kdc[31456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713000, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:40.162745 31693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:40.142638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:52271 (local address 127.27.185.126:41617)
0929 20:16:40.143063 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:40.143067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:40.144000 (+ 933us) server_negotiation.cc:407] Connection header received
0929 20:16:40.145822 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:40.145827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:40.145899 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:40.146000 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:40.148167 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:40.149154 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:40.151581 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:40.151981 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:40.152082 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:40.156413 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:40.156421 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:40.156427 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:40.156472 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:40.159456 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:40.160412 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:40.160415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:40.160418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:40.160530 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:40.161024 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:40.161028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:40.161030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:40.161283 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:40.161390 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:40.162318 (+ 928us) server_negotiation.cc:299] Negotiation successful
0929 20:16:40.162491 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":225,"threads_started":1}
I20260929 20:16:40.164494 31694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:40.143118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41617 (local address 127.27.185.65:52271)
0929 20:16:40.143836 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:40.143874 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:40.145526 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:40.146299 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:40.146311 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:40.147109 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:40.147983 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:40.148006 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:40.149326 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:40.149330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:40.151410 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:40.151431 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:40.152217 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:40.152226 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:40.152571 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:40.153318 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:40.153360 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:40.156133 (+ 2773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:40.159605 (+ 3472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:40.159618 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:40.159639 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:40.160294 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:40.160636 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:40.160643 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:40.160650 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:40.160910 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:40.161405 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:40.161416 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:40.161919 (+ 503us) client_negotiation.cc:769] Sending connection context
0929 20:16:40.162239 (+ 320us) client_negotiation.cc:240] Negotiation successful
0929 20:16:40.162879 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":318,"threads_started":1}
I20260929 20:16:40.166177 31691 heartbeater.cc:347] Connected to a master server at 127.27.185.126:41617
I20260929 20:16:40.166522 31691 heartbeater.cc:464] Registering TS with master...
I20260929 20:16:40.167552 31691 heartbeater.cc:511] Master 127.27.185.126:41617 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.185.66@KRBTEST.COM; defaulting to no policy
I20260929 20:16:40.169973 31501 ts_manager.cc:194] Registered new tserver with Master: ddd0906b46cc41fd86d22a922035b461 (127.27.185.65:41393)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.66@KRBTEST.COM" created.
I20260929 20:16:40.171667 31501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.65@KRBTEST.COM'} at 127.27.185.65:52271
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
I20260929 20:16:40.194974 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:0
--local_ip_for_outbound_sockets=127.27.185.66
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:41617
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:34791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:40.526535 31697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:40.526813 31697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:40.526904 31697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:40.534440 31697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:40.534548 31697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:40.534660 31697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:16:40.544497 31697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:34791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:41617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:40.545794 31697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:40.547049 31697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:40.560961 31705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:40.560931 31703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:40.561010 31702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:40.561707 31697 server_base.cc:1034] running on GCE node
I20260929 20:16:40.562268 31697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:40.563149 31697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:40.564342 31697 hybrid_clock.cc:648] HybridClock initialized: now 1790713000564266 us; error 75 us; skew 500 ppm
I20260929 20:16:40.564620 31697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:40 dist-test-slave-4cx3 krb5kdc[31456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713000, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:40.567811 31697 init.cc:377] Logged in from keytab as kudu/127.27.185.66@KRBTEST.COM (short username kudu)
I20260929 20:16:40.569234 31697 webserver.cc:460] Webserver started at http://127.27.185.66:35461/ using document root <none> and password file <none>
I20260929 20:16:40.569964 31697 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:40.570065 31697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:40.570307 31697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:40.573240 31697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/data/instance:
uuid: "b8a182a11510445ebb1ddfa8dc681974"
format_stamp: "Formatted at 2026-09-29 20:16:40 on dist-test-slave-4cx3"
server_key: "61c83917f894a54334f89fe2a3a5ee86"
server_key_iv: "39d7eb8aa3fda386a268ff5d784ab2ba"
server_key_version: "encryptionkey@0"
I20260929 20:16:40.573992 31697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance:
uuid: "b8a182a11510445ebb1ddfa8dc681974"
format_stamp: "Formatted at 2026-09-29 20:16:40 on dist-test-slave-4cx3"
server_key: "61c83917f894a54334f89fe2a3a5ee86"
server_key_iv: "39d7eb8aa3fda386a268ff5d784ab2ba"
server_key_version: "encryptionkey@0"
I20260929 20:16:40.579082 31697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:16:40.582594 31712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:40.583974 31697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:16:40.584141 31697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "b8a182a11510445ebb1ddfa8dc681974"
format_stamp: "Formatted at 2026-09-29 20:16:40 on dist-test-slave-4cx3"
server_key: "61c83917f894a54334f89fe2a3a5ee86"
server_key_iv: "39d7eb8aa3fda386a268ff5d784ab2ba"
server_key_version: "encryptionkey@0"
I20260929 20:16:40.584335 31697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:40.619216 31697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:40.623155 31697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:40.623479 31697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:40.624440 31697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:16:40.625758 31697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:16:40.625859 31697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:40.626014 31697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:16:40.626085 31697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:40.658682 31697 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:35789
I20260929 20:16:40.658766 31825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:35789 every 8 connection(s)
I20260929 20:16:40.660079 31697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:16:40.661527 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 31697
I20260929 20:16:40.661761 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance
I20260929 20:16:40.662257 28389 external_mini_cluster.cc:1428] Setting key 4be2133dd2be8f691ed2b5c8898fc4ac
Sep 29 20:16:40 dist-test-slave-4cx3 krb5kdc[31456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713000, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.185.67@KRBTEST.COM; defaulting to no policy
I20260929 20:16:40.683203 31693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:40.662875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:55099 (local address 127.27.185.126:41617)
0929 20:16:40.663115 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:40.663120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:40.664603 (+ 1483us) server_negotiation.cc:407] Connection header received
0929 20:16:40.666403 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:40.666409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:40.666497 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:40.666615 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:40.668194 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:40.668906 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:40.671108 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:40.671918 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:40.672061 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:40.676002 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:40.676011 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:40.676019 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:40.676067 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:40.679954 (+ 3887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:40.680737 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:40.680744 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:40.680751 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:40.680939 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:40.681349 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:40.681355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:40.681358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:40.681746 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:40.681950 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:40.682596 (+ 646us) server_negotiation.cc:299] Negotiation successful
0929 20:16:40.682828 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 20:16:40.684357 31828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:40.663554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41617 (local address 127.27.185.66:55099)
0929 20:16:40.664406 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:40.664475 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:40.666134 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:40.666858 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:40.666867 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:40.667430 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:40.668017 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:40.668035 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:40.669077 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:40.669081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:40.670964 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:40.670976 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:40.672169 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:40.672177 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:40.672445 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:40.673143 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:40.673171 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:40.675732 (+ 2561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:40.680109 (+ 4377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:40.680117 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:40.680131 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:40.680592 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:40.681049 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:40.681052 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:40.681054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:40.681224 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:40.681874 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:40.681880 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:40.682204 (+ 324us) client_negotiation.cc:769] Sending connection context
0929 20:16:40.682518 (+ 314us) client_negotiation.cc:240] Negotiation successful
0929 20:16:40.683076 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":324,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.67@KRBTEST.COM" created.
I20260929 20:16:40.686044 31826 heartbeater.cc:347] Connected to a master server at 127.27.185.126:41617
I20260929 20:16:40.686360 31826 heartbeater.cc:464] Registering TS with master...
I20260929 20:16:40.687163 31826 heartbeater.cc:511] Master 127.27.185.126:41617 requested a full tablet report, sending...
I20260929 20:16:40.688807 31501 ts_manager.cc:194] Registered new tserver with Master: b8a182a11510445ebb1ddfa8dc681974 (127.27.185.66:35789)
I20260929 20:16:40.689734 31501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.66@KRBTEST.COM'} at 127.27.185.66:55099
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
I20260929 20:16:40.711145 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:0
--local_ip_for_outbound_sockets=127.27.185.67
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:41617
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:34791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:41.029491 31831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:41.029788 31831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:41.030124 31831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:41.037708 31831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:41.037832 31831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:41.038043 31831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:16:41.047744 31831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:34791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:41617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:41.049086 31831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:41.050431 31831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:41.063854 31837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:41.063936 31839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:41.063956 31836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:41.064875 31831 server_base.cc:1034] running on GCE node
I20260929 20:16:41.065515 31831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:41.066475 31831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:41.067709 31831 hybrid_clock.cc:648] HybridClock initialized: now 1790713001067658 us; error 38 us; skew 500 ppm
I20260929 20:16:41.068070 31831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:41 dist-test-slave-4cx3 krb5kdc[31456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713001, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:41.071355 31831 init.cc:377] Logged in from keytab as kudu/127.27.185.67@KRBTEST.COM (short username kudu)
I20260929 20:16:41.072891 31831 webserver.cc:460] Webserver started at http://127.27.185.67:44745/ using document root <none> and password file <none>
I20260929 20:16:41.073563 31831 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:41.073662 31831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:41.073961 31831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:41.076965 31831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/data/instance:
uuid: "99b46cf6b618423ab43b224765df348c"
format_stamp: "Formatted at 2026-09-29 20:16:41 on dist-test-slave-4cx3"
server_key: "ba463f8b7fab23805329528c0d40fffd"
server_key_iv: "c36374d0500e8da9da100040478ce887"
server_key_version: "encryptionkey@0"
I20260929 20:16:41.077719 31831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance:
uuid: "99b46cf6b618423ab43b224765df348c"
format_stamp: "Formatted at 2026-09-29 20:16:41 on dist-test-slave-4cx3"
server_key: "ba463f8b7fab23805329528c0d40fffd"
server_key_iv: "c36374d0500e8da9da100040478ce887"
server_key_version: "encryptionkey@0"
I20260929 20:16:41.082988 31831 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 20:16:41.086505 31846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:41.088125 31831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 20:16:41.088289 31831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "99b46cf6b618423ab43b224765df348c"
format_stamp: "Formatted at 2026-09-29 20:16:41 on dist-test-slave-4cx3"
server_key: "ba463f8b7fab23805329528c0d40fffd"
server_key_iv: "c36374d0500e8da9da100040478ce887"
server_key_version: "encryptionkey@0"
I20260929 20:16:41.088541 31831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:41.105609 31831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:41.109717 31831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:41.110014 31831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:41.110958 31831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:16:41.112489 31831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:16:41.112560 31831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:41.112689 31831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:16:41.112743 31831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:41.148485 31831 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:40063
I20260929 20:16:41.148526 31958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:40063 every 8 connection(s)
I20260929 20:16:41.149724 31831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:16:41.150148 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 31831
I20260929 20:16:41.150379 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance
I20260929 20:16:41.151001 28389 external_mini_cluster.cc:1428] Setting key 906c15a1558109aa790378a6276ad5d7
Sep 29 20:16:41 dist-test-slave-4cx3 krb5kdc[31456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713001, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:41.169960 31693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:41.152542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:55315 (local address 127.27.185.126:41617)
0929 20:16:41.152704 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:41.152708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:41.154127 (+ 1419us) server_negotiation.cc:407] Connection header received
0929 20:16:41.156087 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:41.156093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:41.156208 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:41.156326 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:41.157771 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:41.158537 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:41.160152 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:41.160551 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:41.160676 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:41.164194 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:41.164200 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:41.164203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:41.164236 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:41.166926 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:41.167707 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:41.167716 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:41.167720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:41.167844 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:41.168254 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:41.168258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:41.168262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:41.168539 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:41.168690 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:41.169383 (+ 693us) server_negotiation.cc:299] Negotiation successful
0929 20:16:41.169610 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 20:16:41.170900 31961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:41.153117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41617 (local address 127.27.185.67:55315)
0929 20:16:41.153961 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:41.154012 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:41.155819 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:41.156561 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:41.156570 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:41.157060 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:41.157610 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:41.157626 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:41.158690 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:41.158700 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:41.160002 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:41.160014 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:41.160790 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:41.160797 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:41.161019 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:41.161678 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:41.161701 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:41.163971 (+ 2270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:41.167069 (+ 3098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:41.167078 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:41.167094 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:41.167587 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:41.167950 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:41.167956 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:41.167960 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:41.168139 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:41.168661 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:41.168668 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:41.169018 (+ 350us) client_negotiation.cc:769] Sending connection context
0929 20:16:41.169282 (+ 264us) client_negotiation.cc:240] Negotiation successful
0929 20:16:41.169803 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":308,"threads_started":1}
I20260929 20:16:41.172314 31959 heartbeater.cc:347] Connected to a master server at 127.27.185.126:41617
I20260929 20:16:41.172602 31959 heartbeater.cc:464] Registering TS with master...
I20260929 20:16:41.173318 31959 heartbeater.cc:511] Master 127.27.185.126:41617 requested a full tablet report, sending...
I20260929 20:16:41.174702 31501 ts_manager.cc:194] Registered new tserver with Master: 99b46cf6b618423ab43b224765df348c (127.27.185.67:40063)
I20260929 20:16:41.175467 31691 heartbeater.cc:503] Master 127.27.185.126:41617 was elected leader, sending a full tablet report...
I20260929 20:16:41.175607 31501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.67@KRBTEST.COM'} at 127.27.185.67:55315
I20260929 20:16:41.178647 28389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 20:16:41 dist-test-slave-4cx3 krb5kdc[31456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713001, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 20:16:41 dist-test-slave-4cx3 krb5kdc[31456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713001, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:41.225919 31693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:41.211785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41166 (local address 127.27.185.126:41617)
0929 20:16:41.211957 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:41.211962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:41.212499 (+ 537us) server_negotiation.cc:407] Connection header received
0929 20:16:41.212696 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:41.212702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:41.212793 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:41.212897 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:41.214282 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:41.215013 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:41.216324 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:41.216747 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:41.216883 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:41.220669 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:41.220674 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:41.220677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:41.220715 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:41.223292 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:41.223991 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:41.223997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:41.224000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:41.224132 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:41.224515 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:41.224520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:41.224524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:41.224846 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:41.225115 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:41.225393 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 20:16:41.225608 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 20:16:41.226508 31501 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41166
I20260929 20:16:41.228752 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 31563
Sep 29 20:16:41 dist-test-slave-4cx3 krb5kdc[31456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713001, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.185.65@KRBTEST.COM
I20260929 20:16:41.243209 31970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:41.229630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39258 (local address 127.27.185.65:41393)
0929 20:16:41.230207 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:41.230213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:41.230237 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:16:41.230291 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:41.230297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:41.230506 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:41.230627 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:41.231784 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:41.232767 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:41.233879 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:41.234423 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:41.234517 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:41.238126 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:41.238137 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:41.238150 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:41.238188 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:41.240948 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:41.241547 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:41.241554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:41.241556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:41.241673 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:41.241961 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:41.241964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:41.241966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:41.242292 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:41.242529 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:41.242735 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 20:16:41.242922 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":271,"threads_started":1}
W20260929 20:16:41.244472 31686 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39258
W20260929 20:16:41.245843 28389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 31563: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:16:41.246055 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 31563
I20260929 20:16:41.262069 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 31697
Sep 29 20:16:41 dist-test-slave-4cx3 krb5kdc[31456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713001, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.185.66@KRBTEST.COM
I20260929 20:16:41.278259 31971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:41.263053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56354 (local address 127.27.185.66:35789)
0929 20:16:41.263670 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:41.263682 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:41.263715 (+ 33us) server_negotiation.cc:407] Connection header received
0929 20:16:41.263796 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:41.263808 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:41.264074 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:41.264255 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:41.265467 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:41.266467 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:41.267852 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:41.268415 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:41.268567 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:41.272690 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:41.272701 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:41.272716 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:41.272752 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:41.275466 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:41.276127 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:41.276140 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:41.276143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:41.276269 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:41.276628 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:41.276635 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:41.276638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:41.276973 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:41.277266 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:41.277603 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 20:16:41.277881 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":234,"threads_started":1}
W20260929 20:16:41.279595 31821 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56354
W20260929 20:16:41.280696 28389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 31697: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:16:41.280886 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 31697
I20260929 20:16:41.296413 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 31831
Sep 29 20:16:41 dist-test-slave-4cx3 krb5kdc[31456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713001, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.185.67@KRBTEST.COM
I20260929 20:16:41.314357 31972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:41.297434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49596 (local address 127.27.185.67:40063)
0929 20:16:41.298212 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:41.298219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:41.298249 (+ 30us) server_negotiation.cc:407] Connection header received
0929 20:16:41.298314 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:41.298320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:41.298534 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:41.298708 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:41.300076 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:41.301214 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:41.302672 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:41.303237 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:41.303370 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:41.308041 (+ 4671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:41.308053 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:41.308069 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:41.308105 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:41.310996 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:41.311964 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:41.311977 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:41.311981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:41.312128 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:41.312546 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:41.312552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:41.312556 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:41.312994 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:41.313329 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:41.313670 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 20:16:41.313982 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":411,"threads_started":1}
W20260929 20:16:41.315809 31954 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49596
W20260929 20:16:41.316874 28389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 31831: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:16:41.317005 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 31831
I20260929 20:16:41.332448 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 31469
I20260929 20:16:41.333626 31531 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:16:41.435912 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 31469
2026-09-29T20:16:41Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3668 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:16:41 dist-test-slave-4cx3 krb5kdc[31976](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:16:41 dist-test-slave-4cx3 krb5kdc[31976](info): set up 2 sockets
Sep 29 20:16:41 dist-test-slave-4cx3 krb5kdc[31976](info): commencing operation
krb5kdc: starting...
W20260929 20:16:42.575636 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:16:42 dist-test-slave-4cx3 krb5kdc[31976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:16:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:16:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
I20260929 20:16:42.778748 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:33317
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:46173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:33317
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:43.102316 31989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:43.102641 31989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:43.102830 31989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:43.110523 31989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:16:43.110626 31989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:43.110730 31989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:43.110816 31989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:16:43.110888 31989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:16:43.120834 31989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:46173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:33317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:33317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:43.122192 31989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:43.123642 31989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:43.135540 31994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:43.135643 31997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:43.135799 31995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:43.140520 31989 server_base.cc:1034] running on GCE node
I20260929 20:16:43.141428 31989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:43.142980 31989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:43.144220 31989 hybrid_clock.cc:648] HybridClock initialized: now 1790713003144171 us; error 59 us; skew 500 ppm
I20260929 20:16:43.144600 31989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:43 dist-test-slave-4cx3 krb5kdc[31976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713003, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:43.148085 31989 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:16:43.149596 31989 webserver.cc:460] Webserver started at http://127.27.185.126:45825/ using document root <none> and password file <none>
I20260929 20:16:43.150322 31989 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:43.150467 31989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:43.150786 31989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:43.153918 31989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "793c4bbcffed4f01995a8bdbb3079136"
format_stamp: "Formatted at 2026-09-29 20:16:43 on dist-test-slave-4cx3"
server_key: "308704496ef6dc1b71924a645a8a0ee4"
server_key_iv: "a3ef7968ff2580dbb5ab81b192d62602"
server_key_version: "encryptionkey@0"
I20260929 20:16:43.154752 31989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "793c4bbcffed4f01995a8bdbb3079136"
format_stamp: "Formatted at 2026-09-29 20:16:43 on dist-test-slave-4cx3"
server_key: "308704496ef6dc1b71924a645a8a0ee4"
server_key_iv: "a3ef7968ff2580dbb5ab81b192d62602"
server_key_version: "encryptionkey@0"
I20260929 20:16:43.160678 31989 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 20:16:43.164208 32004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:43.165843 31989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:16:43.166052 31989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "793c4bbcffed4f01995a8bdbb3079136"
format_stamp: "Formatted at 2026-09-29 20:16:43 on dist-test-slave-4cx3"
server_key: "308704496ef6dc1b71924a645a8a0ee4"
server_key_iv: "a3ef7968ff2580dbb5ab81b192d62602"
server_key_version: "encryptionkey@0"
I20260929 20:16:43.166313 31989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:43.194262 31989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:43.198732 31989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:43.198985 31989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:43.219666 31989 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:33317
I20260929 20:16:43.219672 32055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:33317 every 8 connection(s)
I20260929 20:16:43.221122 31989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:16:43.225080 32056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:43.227887 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 31989
I20260929 20:16:43.228113 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:16:43.228765 28389 external_mini_cluster.cc:1428] Setting key 1aad2e6344dcf6315bb8604e70a024ce
I20260929 20:16:43.238454 32056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 793c4bbcffed4f01995a8bdbb3079136: Bootstrap starting.
I20260929 20:16:43.242760 32056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 793c4bbcffed4f01995a8bdbb3079136: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:43.244232 32056 log.cc:824] T 00000000000000000000000000000000 P 793c4bbcffed4f01995a8bdbb3079136: Log is configured to *not* fsync() on all Append() calls
Sep 29 20:16:43 dist-test-slave-4cx3 krb5kdc[31976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:43.247480 32056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 793c4bbcffed4f01995a8bdbb3079136: No bootstrap required, opened a new log
I20260929 20:16:43.255300 32059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:43.230961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48838 (local address 127.27.185.126:33317)
0929 20:16:43.231811 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:43.231824 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:43.231865 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:16:43.233554 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:43.233577 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:43.233998 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:43.234339 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:43.236344 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:43.237673 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:43.239600 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:43.240325 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:43.240487 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:43.245849 (+ 5362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:43.245858 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:43.245874 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:43.245916 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:43.249928 (+ 4012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:43.250790 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:43.250800 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:43.250811 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:43.251072 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:43.251518 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:43.251523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:43.251528 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:43.252360 (+ 832us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:43.252704 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:43.253163 (+ 459us) server_negotiation.cc:299] Negotiation successful
0929 20:16:43.253700 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":411,"threads_started":1}
I20260929 20:16:43.258706 32056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 793c4bbcffed4f01995a8bdbb3079136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793c4bbcffed4f01995a8bdbb3079136" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33317 } }
I20260929 20:16:43.259068 32056 raft_consensus.cc:374] T 00000000000000000000000000000000 P 793c4bbcffed4f01995a8bdbb3079136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:43.259150 32056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 793c4bbcffed4f01995a8bdbb3079136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793c4bbcffed4f01995a8bdbb3079136, State: Initialized, Role: FOLLOWER
I20260929 20:16:43.259800 32056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 793c4bbcffed4f01995a8bdbb3079136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793c4bbcffed4f01995a8bdbb3079136" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33317 } }
I20260929 20:16:43.260023 32056 raft_consensus.cc:388] T 00000000000000000000000000000000 P 793c4bbcffed4f01995a8bdbb3079136 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:16:43.260124 32056 raft_consensus.cc:482] T 00000000000000000000000000000000 P 793c4bbcffed4f01995a8bdbb3079136 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:16:43.260251 32056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 793c4bbcffed4f01995a8bdbb3079136 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:43.262993 32056 raft_consensus.cc:504] T 00000000000000000000000000000000 P 793c4bbcffed4f01995a8bdbb3079136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793c4bbcffed4f01995a8bdbb3079136" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33317 } }
I20260929 20:16:43.263563 32056 leader_election.cc:302] T 00000000000000000000000000000000 P 793c4bbcffed4f01995a8bdbb3079136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 793c4bbcffed4f01995a8bdbb3079136; no voters:
I20260929 20:16:43.264154 32056 leader_election.cc:288] T 00000000000000000000000000000000 P 793c4bbcffed4f01995a8bdbb3079136 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:16:43.264266 32061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 793c4bbcffed4f01995a8bdbb3079136 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:16:43.265544 32061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 793c4bbcffed4f01995a8bdbb3079136 [term 1 LEADER]: Becoming Leader. State: Replica: 793c4bbcffed4f01995a8bdbb3079136, State: Running, Role: LEADER
I20260929 20:16:43.266124 32061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 793c4bbcffed4f01995a8bdbb3079136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793c4bbcffed4f01995a8bdbb3079136" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33317 } }
I20260929 20:16:43.266487 32056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 793c4bbcffed4f01995a8bdbb3079136 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:16:43.269474 32063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 793c4bbcffed4f01995a8bdbb3079136 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 793c4bbcffed4f01995a8bdbb3079136. Latest consensus state: current_term: 1 leader_uuid: "793c4bbcffed4f01995a8bdbb3079136" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793c4bbcffed4f01995a8bdbb3079136" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33317 } } }
I20260929 20:16:43.270028 32063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 793c4bbcffed4f01995a8bdbb3079136 [sys.catalog]: This master's current role is: LEADER
I20260929 20:16:43.270850 32062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 793c4bbcffed4f01995a8bdbb3079136 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "793c4bbcffed4f01995a8bdbb3079136" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793c4bbcffed4f01995a8bdbb3079136" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33317 } } }
I20260929 20:16:43.271430 32062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 793c4bbcffed4f01995a8bdbb3079136 [sys.catalog]: This master's current role is: LEADER
I20260929 20:16:43.272276 32066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:16:43.277637 32066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:16:43.287743 32066 catalog_manager.cc:1347] Generated new cluster ID: 5833d9f869be4d46827f6b16d1fa7547
I20260929 20:16:43.287892 32066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:16:43.301051 32066 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:16:43.302529 32066 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:16:43.309470 32066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 793c4bbcffed4f01995a8bdbb3079136: Generated new TSK 0
I20260929 20:16:43.310364 32066 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
I20260929 20:16:43.364162 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:0
--local_ip_for_outbound_sockets=127.27.185.65
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:33317
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:46173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:43.713673 32082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:43.713956 32082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:43.714051 32082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:43.721287 32082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:43.721402 32082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:43.721519 32082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:16:43.731280 32082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:46173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:33317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:43.732611 32082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:43.733903 32082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:43.747411 32088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:43.747443 32090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:43.747479 32087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:43.748266 32082 server_base.cc:1034] running on GCE node
I20260929 20:16:43.748845 32082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:43.749789 32082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:43.750984 32082 hybrid_clock.cc:648] HybridClock initialized: now 1790713003750919 us; error 58 us; skew 500 ppm
I20260929 20:16:43.751266 32082 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:43 dist-test-slave-4cx3 krb5kdc[31976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713003, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:43.754501 32082 init.cc:377] Logged in from keytab as kudu/127.27.185.65@KRBTEST.COM (short username kudu)
I20260929 20:16:43.756415 32082 webserver.cc:460] Webserver started at http://127.27.185.65:40087/ using document root <none> and password file <none>
I20260929 20:16:43.757076 32082 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:43.757172 32082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:43.757419 32082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:43.760317 32082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-0/data/instance:
uuid: "74967ae15dd54032a1bd8e6afbeb3fe7"
format_stamp: "Formatted at 2026-09-29 20:16:43 on dist-test-slave-4cx3"
server_key: "662940340b0ee8d2b084edbfb8e4cc7d"
server_key_iv: "8147d6020cfd7b84c18e6db3caa32d85"
server_key_version: "encryptionkey@0"
I20260929 20:16:43.761047 32082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance:
uuid: "74967ae15dd54032a1bd8e6afbeb3fe7"
format_stamp: "Formatted at 2026-09-29 20:16:43 on dist-test-slave-4cx3"
server_key: "662940340b0ee8d2b084edbfb8e4cc7d"
server_key_iv: "8147d6020cfd7b84c18e6db3caa32d85"
server_key_version: "encryptionkey@0"
I20260929 20:16:43.766081 32082 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:16:43.769567 32097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:43.770952 32082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:16:43.771126 32082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "74967ae15dd54032a1bd8e6afbeb3fe7"
format_stamp: "Formatted at 2026-09-29 20:16:43 on dist-test-slave-4cx3"
server_key: "662940340b0ee8d2b084edbfb8e4cc7d"
server_key_iv: "8147d6020cfd7b84c18e6db3caa32d85"
server_key_version: "encryptionkey@0"
I20260929 20:16:43.771337 32082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:43.814047 32082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:43.818269 32082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:43.818625 32082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:43.819693 32082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:16:43.821053 32082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:16:43.821127 32082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:43.821297 32082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:16:43.821355 32082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:43.854097 32082 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:35423
I20260929 20:16:43.854188 32209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:35423 every 8 connection(s)
I20260929 20:16:43.855455 32082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:16:43.857717 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 32082
I20260929 20:16:43.857918 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance
I20260929 20:16:43.858768 28389 external_mini_cluster.cc:1428] Setting key 4c036a1e2124c2f89aaec79592cee657
Sep 29 20:16:43 dist-test-slave-4cx3 krb5kdc[31976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713003, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.185.66@KRBTEST.COM; defaulting to no policy
I20260929 20:16:43.879951 32212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:43.858237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:44937 (local address 127.27.185.126:33317)
0929 20:16:43.858761 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:43.858767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:43.859888 (+ 1121us) server_negotiation.cc:407] Connection header received
0929 20:16:43.862454 (+ 2566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:43.862459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:43.862546 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:43.862637 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:43.864930 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:43.865675 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:43.868002 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:43.868425 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:43.868565 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:43.872630 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:43.872636 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:43.872640 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:43.872678 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:43.876146 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:43.877282 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:43.877287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:43.877290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:43.877413 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:43.877990 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:43.877994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:43.877996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:43.878283 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:43.878419 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:43.879501 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0929 20:16:43.879680 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":247,"threads_started":1}
I20260929 20:16:43.881060 32213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:43.858856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33317 (local address 127.27.185.65:44937)
0929 20:16:43.859699 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:43.859751 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:43.862128 (+ 2377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:43.862958 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:43.862974 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:43.863849 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:43.864730 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:43.864762 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:43.865846 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:43.865854 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:43.867845 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:43.867859 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:43.868680 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:43.868687 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:43.868932 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:43.869685 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:43.869709 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:43.872350 (+ 2641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:43.876330 (+ 3980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:43.876345 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:43.876369 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:43.877075 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:43.877542 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:43.877549 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:43.877554 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:43.877826 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:43.878446 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:43.878457 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:43.878973 (+ 516us) client_negotiation.cc:769] Sending connection context
0929 20:16:43.879316 (+ 343us) client_negotiation.cc:240] Negotiation successful
0929 20:16:43.879813 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":317,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.66@KRBTEST.COM" created.
I20260929 20:16:43.882917 32210 heartbeater.cc:347] Connected to a master server at 127.27.185.126:33317
I20260929 20:16:43.883231 32210 heartbeater.cc:464] Registering TS with master...
I20260929 20:16:43.884024 32210 heartbeater.cc:511] Master 127.27.185.126:33317 requested a full tablet report, sending...
I20260929 20:16:43.886339 32021 ts_manager.cc:194] Registered new tserver with Master: 74967ae15dd54032a1bd8e6afbeb3fe7 (127.27.185.65:35423)
I20260929 20:16:43.888067 32021 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.65@KRBTEST.COM'} at 127.27.185.65:44937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
I20260929 20:16:43.907351 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:0
--local_ip_for_outbound_sockets=127.27.185.66
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:33317
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:46173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:44.235975 32216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:44.236327 32216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:44.236536 32216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:44.243945 32216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:44.244109 32216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:44.244299 32216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:16:44.254247 32216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:46173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:33317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:44.255738 32216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:44.257095 32216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:44.270893 32224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:44.270972 32222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:44.271149 32221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:44.272233 32216 server_base.cc:1034] running on GCE node
I20260929 20:16:44.272836 32216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:44.274044 32216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:44.275257 32216 hybrid_clock.cc:648] HybridClock initialized: now 1790713004275212 us; error 40 us; skew 500 ppm
I20260929 20:16:44.275657 32216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:44 dist-test-slave-4cx3 krb5kdc[31976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713004, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:44.278939 32216 init.cc:377] Logged in from keytab as kudu/127.27.185.66@KRBTEST.COM (short username kudu)
I20260929 20:16:44.280414 32216 webserver.cc:460] Webserver started at http://127.27.185.66:38565/ using document root <none> and password file <none>
I20260929 20:16:44.281133 32216 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:44.281277 32216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:44.281602 32216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:44.284649 32216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-1/data/instance:
uuid: "e5ff0588b5c3422faf148a817e1006a5"
format_stamp: "Formatted at 2026-09-29 20:16:44 on dist-test-slave-4cx3"
server_key: "1e8b66570ad4e2a04ec1ecb247d71bba"
server_key_iv: "68f91e0b74416649815b3f4501097186"
server_key_version: "encryptionkey@0"
I20260929 20:16:44.285445 32216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance:
uuid: "e5ff0588b5c3422faf148a817e1006a5"
format_stamp: "Formatted at 2026-09-29 20:16:44 on dist-test-slave-4cx3"
server_key: "1e8b66570ad4e2a04ec1ecb247d71bba"
server_key_iv: "68f91e0b74416649815b3f4501097186"
server_key_version: "encryptionkey@0"
I20260929 20:16:44.290741 32216 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:16:44.294260 32231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:44.295629 32216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:16:44.295804 32216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "e5ff0588b5c3422faf148a817e1006a5"
format_stamp: "Formatted at 2026-09-29 20:16:44 on dist-test-slave-4cx3"
server_key: "1e8b66570ad4e2a04ec1ecb247d71bba"
server_key_iv: "68f91e0b74416649815b3f4501097186"
server_key_version: "encryptionkey@0"
I20260929 20:16:44.296016 32216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:44.323597 32216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:44.328382 32216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:44.328693 32216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:44.329684 32216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:16:44.331039 32216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:16:44.331143 32216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:44.331298 32216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:16:44.331370 32216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:44.365285 32216 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:45169
I20260929 20:16:44.365309 32343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:45169 every 8 connection(s)
I20260929 20:16:44.366801 32216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:16:44.368063 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 32216
I20260929 20:16:44.368239 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance
I20260929 20:16:44.368819 28389 external_mini_cluster.cc:1428] Setting key 34a14c7d20fec88a64ebc6986dfd3190
Sep 29 20:16:44 dist-test-slave-4cx3 krb5kdc[31976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713004, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.185.67@KRBTEST.COM; defaulting to no policy
I20260929 20:16:44.389560 32212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:44.369847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:37977 (local address 127.27.185.126:33317)
0929 20:16:44.370092 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:44.370097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:44.371625 (+ 1528us) server_negotiation.cc:407] Connection header received
0929 20:16:44.373625 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:44.373630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:44.373720 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:44.373841 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:44.375632 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:44.376764 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:44.378418 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:44.379224 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:44.379371 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:44.383256 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:44.383266 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:44.383275 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:44.383327 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:44.386352 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:44.387140 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:44.387144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:44.387146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:44.387272 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:44.387740 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:44.387745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:44.387748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:44.388083 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:44.388242 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:44.388949 (+ 707us) server_negotiation.cc:299] Negotiation successful
0929 20:16:44.389186 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.67@KRBTEST.COM" created.
I20260929 20:16:44.390594 32346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:44.370571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33317 (local address 127.27.185.66:37977)
0929 20:16:44.371476 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:44.371522 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:44.373345 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:44.374136 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:44.374150 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:44.374750 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:44.375451 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:44.375474 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:44.376968 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:44.376973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:44.378255 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:44.378272 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:44.379613 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:44.379620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:44.379830 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:44.380615 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:44.380648 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:44.382977 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:44.386521 (+ 3544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:44.386530 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:44.386549 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:44.386997 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:44.387403 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:44.387411 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:44.387417 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:44.387612 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:44.388213 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:44.388222 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:44.388577 (+ 355us) client_negotiation.cc:769] Sending connection context
0929 20:16:44.388847 (+ 270us) client_negotiation.cc:240] Negotiation successful
0929 20:16:44.389428 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":327,"threads_started":1}
I20260929 20:16:44.392191 32344 heartbeater.cc:347] Connected to a master server at 127.27.185.126:33317
I20260929 20:16:44.392542 32344 heartbeater.cc:464] Registering TS with master...
I20260929 20:16:44.393432 32344 heartbeater.cc:511] Master 127.27.185.126:33317 requested a full tablet report, sending...
I20260929 20:16:44.395103 32021 ts_manager.cc:194] Registered new tserver with Master: e5ff0588b5c3422faf148a817e1006a5 (127.27.185.66:45169)
I20260929 20:16:44.395972 32021 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.66@KRBTEST.COM'} at 127.27.185.66:37977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
I20260929 20:16:44.414675 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:0
--local_ip_for_outbound_sockets=127.27.185.67
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:33317
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:46173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:44.746768 32349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:44.747081 32349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:44.747177 32349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:44.754721 32349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:44.754844 32349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:44.754959 32349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:16:44.765100 32349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:46173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:33317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:44.766486 32349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:44.767953 32349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:44.781975 32354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:44.781921 32355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:44.782017 32357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:44.782863 32349 server_base.cc:1034] running on GCE node
I20260929 20:16:44.783574 32349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:44.784528 32349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:44.785729 32349 hybrid_clock.cc:648] HybridClock initialized: now 1790713004785663 us; error 57 us; skew 500 ppm
I20260929 20:16:44.786059 32349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:44 dist-test-slave-4cx3 krb5kdc[31976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713004, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:44.789235 32349 init.cc:377] Logged in from keytab as kudu/127.27.185.67@KRBTEST.COM (short username kudu)
I20260929 20:16:44.790817 32349 webserver.cc:460] Webserver started at http://127.27.185.67:43243/ using document root <none> and password file <none>
I20260929 20:16:44.791585 32349 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:44.791718 32349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:44.792052 32349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:44.795015 32349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-2/data/instance:
uuid: "9d9b103cd1c942cda53a857654620690"
format_stamp: "Formatted at 2026-09-29 20:16:44 on dist-test-slave-4cx3"
server_key: "9107b578bfbf592fdfbf5e5c67acb25f"
server_key_iv: "737b4549fabf27eee45ad1c50abfaa64"
server_key_version: "encryptionkey@0"
I20260929 20:16:44.795838 32349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance:
uuid: "9d9b103cd1c942cda53a857654620690"
format_stamp: "Formatted at 2026-09-29 20:16:44 on dist-test-slave-4cx3"
server_key: "9107b578bfbf592fdfbf5e5c67acb25f"
server_key_iv: "737b4549fabf27eee45ad1c50abfaa64"
server_key_version: "encryptionkey@0"
I20260929 20:16:44.801048 32349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:16:44.804515 32364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:44.805856 32349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:16:44.806046 32349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "9d9b103cd1c942cda53a857654620690"
format_stamp: "Formatted at 2026-09-29 20:16:44 on dist-test-slave-4cx3"
server_key: "9107b578bfbf592fdfbf5e5c67acb25f"
server_key_iv: "737b4549fabf27eee45ad1c50abfaa64"
server_key_version: "encryptionkey@0"
I20260929 20:16:44.806257 32349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:44.830972 32349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:44.835278 32349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:44.835604 32349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:44.836664 32349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:16:44.838045 32349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:16:44.838117 32349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:44.838263 32349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:16:44.838323 32349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:44.872716 32349 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:38157
I20260929 20:16:44.872756 32476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:38157 every 8 connection(s)
I20260929 20:16:44.874094 32349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:16:44.874806 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 32349
I20260929 20:16:44.875015 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance
I20260929 20:16:44.875777 28389 external_mini_cluster.cc:1428] Setting key bb2d9f5295957305f59574764d869875
Sep 29 20:16:44 dist-test-slave-4cx3 krb5kdc[31976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713004, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:44.891683 32210 heartbeater.cc:503] Master 127.27.185.126:33317 was elected leader, sending a full tablet report...
I20260929 20:16:44.894487 32212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:44.877135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:60447 (local address 127.27.185.126:33317)
0929 20:16:44.877359 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:44.877365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:44.878641 (+ 1276us) server_negotiation.cc:407] Connection header received
0929 20:16:44.880602 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:44.880606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:44.880677 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:44.880760 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:44.882232 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:44.882982 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:44.884531 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:44.884923 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:44.885019 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:44.888339 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:44.888344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:44.888347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:44.888388 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:44.891076 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:44.891980 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:44.891985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:44.891990 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:44.892198 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:44.892744 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:44.892748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:44.892752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:44.893110 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:44.893264 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:44.894069 (+ 805us) server_negotiation.cc:299] Negotiation successful
0929 20:16:44.894244 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:16:44.895531 32479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:44.877682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33317 (local address 127.27.185.67:60447)
0929 20:16:44.878500 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:44.878546 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:44.880331 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:44.880972 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:44.880979 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:44.881495 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:44.882071 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:44.882088 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:44.883109 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:44.883113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:44.884400 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:44.884412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:44.885112 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:44.885116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:44.885304 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:44.885919 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:44.885942 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:44.888125 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:44.891257 (+ 3132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:44.891268 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:44.891290 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:44.891843 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:44.892335 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:44.892341 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:44.892345 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:44.892612 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:44.893286 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:44.893297 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:44.893694 (+ 397us) client_negotiation.cc:769] Sending connection context
0929 20:16:44.893946 (+ 252us) client_negotiation.cc:240] Negotiation successful
0929 20:16:44.894363 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":353,"threads_started":1}
I20260929 20:16:44.896984 32477 heartbeater.cc:347] Connected to a master server at 127.27.185.126:33317
I20260929 20:16:44.897267 32477 heartbeater.cc:464] Registering TS with master...
I20260929 20:16:44.897936 32477 heartbeater.cc:511] Master 127.27.185.126:33317 requested a full tablet report, sending...
I20260929 20:16:44.899564 32021 ts_manager.cc:194] Registered new tserver with Master: 9d9b103cd1c942cda53a857654620690 (127.27.185.67:38157)
I20260929 20:16:44.900450 32021 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.67@KRBTEST.COM'} at 127.27.185.67:60447
I20260929 20:16:44.903534 28389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 20:16:44 dist-test-slave-4cx3 krb5kdc[31976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 20:16:44 dist-test-slave-4cx3 krb5kdc[31976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.65@KRBTEST.COM
I20260929 20:16:44.950469 32484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:44.935983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46876 (local address 127.27.185.65:35423)
0929 20:16:44.936689 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:44.936697 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:44.936724 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:16:44.936795 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:44.936802 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:44.937023 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:44.937172 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:44.938542 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:44.939637 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:44.940831 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:44.941396 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:44.941496 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:44.945359 (+ 3863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:44.945371 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:44.945385 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:44.945423 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:44.948139 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:44.948730 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:44.948736 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:44.948739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:44.948856 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:44.949188 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:44.949191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:44.949193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:44.949529 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:44.949756 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:44.949999 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 20:16:44.950190 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":375,"threads_started":1}
Sep 29 20:16:44 dist-test-slave-4cx3 krb5kdc[31976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:44.967788 32212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:44.954651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48840 (local address 127.27.185.126:33317)
0929 20:16:44.954861 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:44.954866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:44.955062 (+ 196us) server_negotiation.cc:407] Connection header received
0929 20:16:44.955188 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:44.955192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:44.955252 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:44.955331 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:44.956501 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:44.957205 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:44.958403 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:44.958831 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:44.958927 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:44.962684 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:44.962691 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:44.962694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:44.962731 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:44.965375 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:44.965965 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:44.965969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:44.965971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:44.966102 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:44.966403 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:44.966407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:44.966409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:44.966737 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:44.966968 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:44.967222 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 20:16:44.967431 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260929 20:16:44.968405 32021 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48840
I20260929 20:16:44.969416 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 32082
I20260929 20:16:44.981760 32484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:44.970370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46888 (local address 127.27.185.65:35423)
0929 20:16:44.970567 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:44.970572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:44.970758 (+ 186us) server_negotiation.cc:407] Connection header received
0929 20:16:44.970953 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:44.970958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:44.971030 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:44.971107 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:44.972290 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:44.972999 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:44.974163 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:44.974703 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:44.974801 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:44.977208 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:44.977212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:44.977215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:44.977247 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:44.979662 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:44.980268 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:44.980272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:44.980274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:44.980385 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:44.980681 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:44.980684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:44.980686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:44.980913 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:44.981090 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:44.981371 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 20:16:44.981530 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 20:16:44.982357 32205 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:16:45.065773 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 32082
I20260929 20:16:45.082329 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 32216
Sep 29 20:16:45 dist-test-slave-4cx3 krb5kdc[31976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.66@KRBTEST.COM
I20260929 20:16:45.102602 32489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:45.083305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38606 (local address 127.27.185.66:45169)
0929 20:16:45.083923 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:45.083930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:45.083954 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:16:45.084026 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:45.084032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:45.084263 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:45.084444 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:45.085724 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:45.086747 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:45.088157 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:45.088700 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:45.088856 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:45.097008 (+ 8152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:45.097023 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:45.097038 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:45.097076 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:45.099945 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:45.100729 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:45.100736 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:45.100738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:45.100857 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:45.101183 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:45.101186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:45.101188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:45.101521 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:45.101751 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:45.102090 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 20:16:45.102294 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":371,"threads_started":1}
I20260929 20:16:45.104120 32339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:16:45.186532 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 32216
I20260929 20:16:45.202518 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 32349
Sep 29 20:16:45 dist-test-slave-4cx3 krb5kdc[31976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.67@KRBTEST.COM
I20260929 20:16:45.218446 32491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:45.203582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59712 (local address 127.27.185.67:38157)
0929 20:16:45.204193 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:45.204200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:45.204229 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:16:45.204289 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:45.204295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:45.204536 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:45.204666 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:45.205895 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:45.206905 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:45.208176 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:45.208721 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:45.208814 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:45.213230 (+ 4416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:45.213241 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:45.213256 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:45.213292 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:45.216035 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:45.216653 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:45.216659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:45.216662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:45.216789 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:45.217097 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:45.217100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:45.217102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:45.217414 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:45.217645 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:45.217870 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 20:16:45.218103 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":320,"threads_started":1}
I20260929 20:16:45.219964 32472 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:16:45.306494 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 32349
I20260929 20:16:45.323313 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 31989
I20260929 20:16:45.324625 32051 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:16:45.428814 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 31989
2026-09-29T20:16:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3993 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:16:45 dist-test-slave-4cx3 krb5kdc[32496](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:16:45 dist-test-slave-4cx3 krb5kdc[32496](info): set up 2 sockets
Sep 29 20:16:45 dist-test-slave-4cx3 krb5kdc[32496](info): commencing operation
krb5kdc: starting...
W20260929 20:16:46.559027 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:16:46 dist-test-slave-4cx3 krb5kdc[32496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:16:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:16:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
I20260929 20:16:46.771214 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:42515
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:42515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:47.095808 32509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:47.096113 32509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:47.096316 32509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:47.103709 32509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:16:47.103852 32509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:47.103938 32509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:47.104039 32509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:16:47.104126 32509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:16:47.115216 32509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:42515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:42515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:47.116675 32509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:47.118132 32509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:47.130034 32515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:47.130116 32514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:47.130112 32517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:47.131067 32509 server_base.cc:1034] running on GCE node
I20260929 20:16:47.131896 32509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:47.133268 32509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:47.134469 32509 hybrid_clock.cc:648] HybridClock initialized: now 1790713007134413 us; error 62 us; skew 500 ppm
I20260929 20:16:47.134848 32509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:47 dist-test-slave-4cx3 krb5kdc[32496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713007, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:47.137928 32509 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:16:47.139122 32509 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:47.139249 32509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:47.139632 32509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:47.142524 32509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "1c3cbc69eac04aeea8a4acb70e8412ee"
format_stamp: "Formatted at 2026-09-29 20:16:47 on dist-test-slave-4cx3"
server_key: "7b41bd555161b0adfd6ca9f9fbf2775c"
server_key_iv: "3a827194a99c85c8fa33ab607d56b718"
server_key_version: "encryptionkey@0"
I20260929 20:16:47.143313 32509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "1c3cbc69eac04aeea8a4acb70e8412ee"
format_stamp: "Formatted at 2026-09-29 20:16:47 on dist-test-slave-4cx3"
server_key: "7b41bd555161b0adfd6ca9f9fbf2775c"
server_key_iv: "3a827194a99c85c8fa33ab607d56b718"
server_key_version: "encryptionkey@0"
I20260929 20:16:47.148296 32509 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 20:16:47.151854 32523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:47.153138 32509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 20:16:47.153319 32509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "1c3cbc69eac04aeea8a4acb70e8412ee"
format_stamp: "Formatted at 2026-09-29 20:16:47 on dist-test-slave-4cx3"
server_key: "7b41bd555161b0adfd6ca9f9fbf2775c"
server_key_iv: "3a827194a99c85c8fa33ab607d56b718"
server_key_version: "encryptionkey@0"
I20260929 20:16:47.153553 32509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:47.171306 32509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:47.175551 32509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:47.175841 32509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:47.194016 32509 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:42515
I20260929 20:16:47.194036 32574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:42515 every 8 connection(s)
I20260929 20:16:47.194929 32509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:16:47.198649 32575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:47.200415 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 32509
I20260929 20:16:47.200642 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:16:47.201264 28389 external_mini_cluster.cc:1428] Setting key 516b977f7b4b9a87d74683d3d1d85d76
I20260929 20:16:47.211683 32575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c3cbc69eac04aeea8a4acb70e8412ee: Bootstrap starting.
I20260929 20:16:47.215508 32575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c3cbc69eac04aeea8a4acb70e8412ee: Neither blocks nor log segments found. Creating new log.
Sep 29 20:16:47 dist-test-slave-4cx3 krb5kdc[32496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:47.216739 32575 log.cc:824] T 00000000000000000000000000000000 P 1c3cbc69eac04aeea8a4acb70e8412ee: Log is configured to *not* fsync() on all Append() calls
I20260929 20:16:47.220304 32575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c3cbc69eac04aeea8a4acb70e8412ee: No bootstrap required, opened a new log
I20260929 20:16:47.226580 32578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:47.203227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49726 (local address 127.27.185.126:42515)
0929 20:16:47.204129 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:47.204144 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:47.204197 (+ 53us) server_negotiation.cc:407] Connection header received
0929 20:16:47.206528 (+ 2331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:47.206555 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:47.207044 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:47.207505 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:47.208880 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:47.210871 (+ 1991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:47.212218 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:47.213211 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:47.213430 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:47.218034 (+ 4604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:47.218046 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:47.218066 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:47.218113 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:47.221906 (+ 3793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:47.222633 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:47.222642 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:47.222651 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:47.222859 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:47.223218 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:47.223224 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:47.223227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:47.224055 (+ 828us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:47.224363 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:47.224712 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 20:16:47.225167 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":335,"threads_started":1}
I20260929 20:16:47.229978 32575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c3cbc69eac04aeea8a4acb70e8412ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3cbc69eac04aeea8a4acb70e8412ee" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 42515 } }
I20260929 20:16:47.230302 32575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c3cbc69eac04aeea8a4acb70e8412ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:47.230369 32575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c3cbc69eac04aeea8a4acb70e8412ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3cbc69eac04aeea8a4acb70e8412ee, State: Initialized, Role: FOLLOWER
I20260929 20:16:47.230916 32575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c3cbc69eac04aeea8a4acb70e8412ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3cbc69eac04aeea8a4acb70e8412ee" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 42515 } }
I20260929 20:16:47.231207 32575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c3cbc69eac04aeea8a4acb70e8412ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:16:47.231350 32575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c3cbc69eac04aeea8a4acb70e8412ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:16:47.231609 32575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c3cbc69eac04aeea8a4acb70e8412ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:47.233752 32575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c3cbc69eac04aeea8a4acb70e8412ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3cbc69eac04aeea8a4acb70e8412ee" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 42515 } }
I20260929 20:16:47.234184 32575 leader_election.cc:302] T 00000000000000000000000000000000 P 1c3cbc69eac04aeea8a4acb70e8412ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c3cbc69eac04aeea8a4acb70e8412ee; no voters:
I20260929 20:16:47.234678 32575 leader_election.cc:288] T 00000000000000000000000000000000 P 1c3cbc69eac04aeea8a4acb70e8412ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:16:47.234828 32580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c3cbc69eac04aeea8a4acb70e8412ee [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:16:47.235515 32580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c3cbc69eac04aeea8a4acb70e8412ee [term 1 LEADER]: Becoming Leader. State: Replica: 1c3cbc69eac04aeea8a4acb70e8412ee, State: Running, Role: LEADER
I20260929 20:16:47.236039 32580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c3cbc69eac04aeea8a4acb70e8412ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3cbc69eac04aeea8a4acb70e8412ee" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 42515 } }
I20260929 20:16:47.236975 32575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c3cbc69eac04aeea8a4acb70e8412ee [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:16:47.239181 32581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c3cbc69eac04aeea8a4acb70e8412ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c3cbc69eac04aeea8a4acb70e8412ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3cbc69eac04aeea8a4acb70e8412ee" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 42515 } } }
I20260929 20:16:47.239133 32582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c3cbc69eac04aeea8a4acb70e8412ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c3cbc69eac04aeea8a4acb70e8412ee. Latest consensus state: current_term: 1 leader_uuid: "1c3cbc69eac04aeea8a4acb70e8412ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3cbc69eac04aeea8a4acb70e8412ee" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 42515 } } }
I20260929 20:16:47.239557 32582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c3cbc69eac04aeea8a4acb70e8412ee [sys.catalog]: This master's current role is: LEADER
I20260929 20:16:47.239560 32581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c3cbc69eac04aeea8a4acb70e8412ee [sys.catalog]: This master's current role is: LEADER
I20260929 20:16:47.240525 32587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:16:47.247256 32587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:16:47.257201 32587 catalog_manager.cc:1347] Generated new cluster ID: e0fc8f8b06de49ad8e15a3aa9f7cc424
I20260929 20:16:47.257334 32587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:16:47.280107 32587 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:16:47.282074 32587 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:16:47.295650 32587 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c3cbc69eac04aeea8a4acb70e8412ee: Generated new TSK 0
I20260929 20:16:47.296768 32587 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
I20260929 20:16:47.355026 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:0
--local_ip_for_outbound_sockets=127.27.185.65
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:42515
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:47.685902 32601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:47.686193 32601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:47.686393 32601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:47.693620 32601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:47.693776 32601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:47.693981 32601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:16:47.703828 32601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:42515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:47.705220 32601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:47.706573 32601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:47.720184 32606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:47.720116 32609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:47.720141 32607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:47.720995 32601 server_base.cc:1034] running on GCE node
I20260929 20:16:47.721547 32601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:47.722466 32601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:47.723697 32601 hybrid_clock.cc:648] HybridClock initialized: now 1790713007723634 us; error 63 us; skew 500 ppm
I20260929 20:16:47.724043 32601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:47 dist-test-slave-4cx3 krb5kdc[32496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713007, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:47.727319 32601 init.cc:377] Logged in from keytab as kudu/127.27.185.65@KRBTEST.COM (short username kudu)
I20260929 20:16:47.728580 32601 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:47.728715 32601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:47.729051 32601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:47.732117 32601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-0/data/instance:
uuid: "eeb9937f42f64596b59fe0ea25b42da2"
format_stamp: "Formatted at 2026-09-29 20:16:47 on dist-test-slave-4cx3"
server_key: "46c30d6c371e88451744922911785a8c"
server_key_iv: "a7213f93d2c35e960c2a6d8aea716955"
server_key_version: "encryptionkey@0"
I20260929 20:16:47.732997 32601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance:
uuid: "eeb9937f42f64596b59fe0ea25b42da2"
format_stamp: "Formatted at 2026-09-29 20:16:47 on dist-test-slave-4cx3"
server_key: "46c30d6c371e88451744922911785a8c"
server_key_iv: "a7213f93d2c35e960c2a6d8aea716955"
server_key_version: "encryptionkey@0"
I20260929 20:16:47.738505 32601 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 20:16:47.742213 32615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:47.743748 32601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 20:16:47.743963 32601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "eeb9937f42f64596b59fe0ea25b42da2"
format_stamp: "Formatted at 2026-09-29 20:16:47 on dist-test-slave-4cx3"
server_key: "46c30d6c371e88451744922911785a8c"
server_key_iv: "a7213f93d2c35e960c2a6d8aea716955"
server_key_version: "encryptionkey@0"
I20260929 20:16:47.744213 32601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:47.761663 32601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:47.765865 32601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:47.766177 32601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:47.767180 32601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:16:47.768633 32601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:16:47.768761 32601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:47.768885 32601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:16:47.768971 32601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:47.801520 32601 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:38505
I20260929 20:16:47.801568 32727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:38505 every 8 connection(s)
I20260929 20:16:47.802407 32601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:16:47.806210 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 32601
I20260929 20:16:47.806432 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance
I20260929 20:16:47.807102 28389 external_mini_cluster.cc:1428] Setting key 6ce927461d34a26f3d6eb8033b5270a6
Sep 29 20:16:47 dist-test-slave-4cx3 krb5kdc[32496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713007, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:47.823150 32730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:47.804922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:55189 (local address 127.27.185.126:42515)
0929 20:16:47.805372 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:47.805376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:47.806201 (+ 825us) server_negotiation.cc:407] Connection header received
0929 20:16:47.808503 (+ 2302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:47.808508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:47.808596 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:47.808719 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:47.810275 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:47.810999 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:47.812648 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:47.813073 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:47.813205 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:47.817386 (+ 4181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:47.817391 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:47.817394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:47.817426 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:47.819986 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:47.820743 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:47.820747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:47.820750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:47.820869 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:47.821317 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:47.821322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:47.821325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:47.821604 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:47.821760 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:47.822506 (+ 746us) server_negotiation.cc:299] Negotiation successful
0929 20:16:47.822751 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":246,"threads_started":1}
I20260929 20:16:47.824249 32731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:47.805353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:42515 (local address 127.27.185.65:55189)
0929 20:16:47.806043 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:47.806088 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:47.808217 (+ 2129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:47.808966 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:47.808981 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:47.809520 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:47.810105 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:47.810127 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:47.811158 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:47.811162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:47.812496 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:47.812507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:47.813310 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:47.813317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:47.813539 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:47.814244 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:47.814279 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:47.817158 (+ 2879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:47.820133 (+ 2975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:47.820143 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:47.820162 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:47.820606 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:47.820988 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:47.820993 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:47.820997 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:47.821192 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:47.821730 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:47.821737 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:47.822113 (+ 376us) client_negotiation.cc:769] Sending connection context
0929 20:16:47.822386 (+ 273us) client_negotiation.cc:240] Negotiation successful
0929 20:16:47.823037 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":273,"threads_started":1}
WARNING: no policy specified for kudu/127.27.185.66@KRBTEST.COM; defaulting to no policy
I20260929 20:16:47.826002 32728 heartbeater.cc:347] Connected to a master server at 127.27.185.126:42515
I20260929 20:16:47.826313 32728 heartbeater.cc:464] Registering TS with master...
I20260929 20:16:47.827073 32728 heartbeater.cc:511] Master 127.27.185.126:42515 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.66@KRBTEST.COM" created.
I20260929 20:16:47.829438 32540 ts_manager.cc:194] Registered new tserver with Master: eeb9937f42f64596b59fe0ea25b42da2 (127.27.185.65:38505)
I20260929 20:16:47.830915 32540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.65@KRBTEST.COM'} at 127.27.185.65:55189
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
I20260929 20:16:47.852747 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:0
--local_ip_for_outbound_sockets=127.27.185.66
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:42515
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:48.181453 32734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:48.181734 32734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:48.181830 32734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:48.189507 32734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:48.189658 32734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:48.189878 32734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:16:48.203972 32734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:42515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:48.205570 32734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:48.207121 32734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:48.221834 32739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:48.221779 32742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:48.221779 32740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:48.222713 32734 server_base.cc:1034] running on GCE node
I20260929 20:16:48.223308 32734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:48.224253 32734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:48.225454 32734 hybrid_clock.cc:648] HybridClock initialized: now 1790713008225400 us; error 34 us; skew 500 ppm
I20260929 20:16:48.225747 32734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:48 dist-test-slave-4cx3 krb5kdc[32496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713008, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:48.228945 32734 init.cc:377] Logged in from keytab as kudu/127.27.185.66@KRBTEST.COM (short username kudu)
I20260929 20:16:48.230108 32734 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:48.230218 32734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:48.230487 32734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:48.233474 32734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-1/data/instance:
uuid: "f3ffdbf04d034b1e89d34aa52659e3f1"
format_stamp: "Formatted at 2026-09-29 20:16:48 on dist-test-slave-4cx3"
server_key: "e781a02c0e9f648aa671305fd510004b"
server_key_iv: "2e634bc9f21779a9911b13eb19d88829"
server_key_version: "encryptionkey@0"
I20260929 20:16:48.234263 32734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance:
uuid: "f3ffdbf04d034b1e89d34aa52659e3f1"
format_stamp: "Formatted at 2026-09-29 20:16:48 on dist-test-slave-4cx3"
server_key: "e781a02c0e9f648aa671305fd510004b"
server_key_iv: "2e634bc9f21779a9911b13eb19d88829"
server_key_version: "encryptionkey@0"
I20260929 20:16:48.239249 32734 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 20:16:48.242661 32748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:48.244148 32734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:16:48.244298 32734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "f3ffdbf04d034b1e89d34aa52659e3f1"
format_stamp: "Formatted at 2026-09-29 20:16:48 on dist-test-slave-4cx3"
server_key: "e781a02c0e9f648aa671305fd510004b"
server_key_iv: "2e634bc9f21779a9911b13eb19d88829"
server_key_version: "encryptionkey@0"
I20260929 20:16:48.244493 32734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:48.290516 32734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:48.295217 32734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:48.295578 32734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:48.296572 32734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:16:48.297955 32734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:16:48.298040 32734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:48.298192 32734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:16:48.298285 32734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:48.331542 32734 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:33201
I20260929 20:16:48.331575 392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:33201 every 8 connection(s)
I20260929 20:16:48.332413 32734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:16:48.342864 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 32734
I20260929 20:16:48.343052 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance
I20260929 20:16:48.343680 28389 external_mini_cluster.cc:1428] Setting key cdab8a0624b54ea08c5b1a75ff3a2a61
Sep 29 20:16:48 dist-test-slave-4cx3 krb5kdc[32496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713008, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:48.353195 395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:48.334869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:35181 (local address 127.27.185.126:42515)
0929 20:16:48.335327 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:48.335334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:48.336325 (+ 991us) server_negotiation.cc:407] Connection header received
0929 20:16:48.338162 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:48.338166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:48.338232 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:48.338326 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:48.339806 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:48.340511 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:48.341950 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:48.342357 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:48.342463 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:48.346606 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:48.346611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:48.346614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:48.346645 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:48.349710 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:48.350670 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:48.350673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:48.350676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:48.350789 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:48.351248 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:48.351251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:48.351253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:48.351569 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:48.351699 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:48.352444 (+ 745us) server_negotiation.cc:299] Negotiation successful
0929 20:16:48.352704 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":223,"threads_started":1}
I20260929 20:16:48.353858 396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:48.335471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:42515 (local address 127.27.185.66:35181)
0929 20:16:48.336161 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:48.336206 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:48.337871 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:48.338526 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:48.338534 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:48.339037 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:48.339654 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:48.339672 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:48.340628 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:48.340632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:48.341830 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:48.341841 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:48.342592 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:48.342600 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:48.342886 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:48.343671 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:48.343711 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:48.346381 (+ 2670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:48.349854 (+ 3473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:48.349867 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:48.349887 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:48.350553 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:48.350894 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:48.350901 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:48.350907 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:48.351140 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:48.351696 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:48.351706 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:48.352071 (+ 365us) client_negotiation.cc:769] Sending connection context
0929 20:16:48.352301 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 20:16:48.352707 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":258,"threads_started":1}
I20260929 20:16:48.355422 393 heartbeater.cc:347] Connected to a master server at 127.27.185.126:42515
I20260929 20:16:48.355756 393 heartbeater.cc:464] Registering TS with master...
I20260929 20:16:48.356552 393 heartbeater.cc:511] Master 127.27.185.126:42515 requested a full tablet report, sending...
I20260929 20:16:48.358276 32540 ts_manager.cc:194] Registered new tserver with Master: f3ffdbf04d034b1e89d34aa52659e3f1 (127.27.185.66:33201)
I20260929 20:16:48.359115 32540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.66@KRBTEST.COM'} at 127.27.185.66:35181
WARNING: no policy specified for kudu/127.27.185.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
I20260929 20:16:48.390838 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:0
--local_ip_for_outbound_sockets=127.27.185.67
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:42515
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:48.713500 399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:48.713796 399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:48.713963 399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:48.721578 399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:48.721684 399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:48.721889 399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:16:48.731941 399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:42515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:48.733310 399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:48.734625 399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:48.748586 405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:48.748667 404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:48.748688 407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:48.749536 399 server_base.cc:1034] running on GCE node
I20260929 20:16:48.750167 399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:48.751127 399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:48.752349 399 hybrid_clock.cc:648] HybridClock initialized: now 1790713008752275 us; error 67 us; skew 500 ppm
I20260929 20:16:48.752688 399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:48 dist-test-slave-4cx3 krb5kdc[32496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713008, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:48.755725 399 init.cc:377] Logged in from keytab as kudu/127.27.185.67@KRBTEST.COM (short username kudu)
I20260929 20:16:48.756991 399 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:48.757122 399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:48.757440 399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:48.760528 399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-2/data/instance:
uuid: "9f6486ecaabd44aabb2385046a619f64"
format_stamp: "Formatted at 2026-09-29 20:16:48 on dist-test-slave-4cx3"
server_key: "d7577f080637604a766af70d5e680f2c"
server_key_iv: "e97e52a21738da701431ac368c8f7c2b"
server_key_version: "encryptionkey@0"
I20260929 20:16:48.761314 399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance:
uuid: "9f6486ecaabd44aabb2385046a619f64"
format_stamp: "Formatted at 2026-09-29 20:16:48 on dist-test-slave-4cx3"
server_key: "d7577f080637604a766af70d5e680f2c"
server_key_iv: "e97e52a21738da701431ac368c8f7c2b"
server_key_version: "encryptionkey@0"
I20260929 20:16:48.766384 399 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 20:16:48.769946 413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:48.771311 399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:16:48.771533 399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "9f6486ecaabd44aabb2385046a619f64"
format_stamp: "Formatted at 2026-09-29 20:16:48 on dist-test-slave-4cx3"
server_key: "d7577f080637604a766af70d5e680f2c"
server_key_iv: "e97e52a21738da701431ac368c8f7c2b"
server_key_version: "encryptionkey@0"
I20260929 20:16:48.771754 399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:48.786011 399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:48.789844 399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:48.790125 399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:48.791025 399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:16:48.792372 399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:16:48.792497 399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:48.792634 399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:16:48.792724 399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:48.826233 399 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:37285
I20260929 20:16:48.826283 525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:37285 every 8 connection(s)
I20260929 20:16:48.827149 399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:16:48.829854 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 399
I20260929 20:16:48.830075 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance
I20260929 20:16:48.830585 28389 external_mini_cluster.cc:1428] Setting key fd7d55222c1d4a605c40dd2774422506
I20260929 20:16:48.834578 32728 heartbeater.cc:503] Master 127.27.185.126:42515 was elected leader, sending a full tablet report...
Sep 29 20:16:48 dist-test-slave-4cx3 krb5kdc[32496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713008, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:48.847146 395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:48.829657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:41847 (local address 127.27.185.126:42515)
0929 20:16:48.829901 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:48.829909 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:48.831231 (+ 1322us) server_negotiation.cc:407] Connection header received
0929 20:16:48.833560 (+ 2329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:48.833565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:48.833674 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:48.833777 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:48.835597 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:48.836464 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:48.838006 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:48.838405 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:48.838559 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:48.841740 (+ 3181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:48.841748 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:48.841758 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:48.841788 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:48.844111 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:48.844936 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:48.844946 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:48.844948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:48.845058 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:48.845489 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:48.845500 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:48.845503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:48.845764 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:48.845922 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:48.846576 (+ 654us) server_negotiation.cc:299] Negotiation successful
0929 20:16:48.846791 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 20:16:48.848172 528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:48.830222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:42515 (local address 127.27.185.67:41847)
0929 20:16:48.831054 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:48.831105 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:48.833274 (+ 2169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:48.834019 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:48.834027 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:48.834770 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:48.835368 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:48.835431 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:48.836620 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:48.836626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:48.837860 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:48.837872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:48.838659 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:48.838667 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:48.838887 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:48.839532 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:48.839557 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:48.841556 (+ 1999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:48.844242 (+ 2686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:48.844250 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:48.844265 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:48.844815 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:48.845180 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:48.845184 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:48.845188 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:48.845382 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:48.845892 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:48.845900 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:48.846222 (+ 322us) client_negotiation.cc:769] Sending connection context
0929 20:16:48.846467 (+ 245us) client_negotiation.cc:240] Negotiation successful
0929 20:16:48.846957 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":297,"threads_started":1}
I20260929 20:16:48.849515 526 heartbeater.cc:347] Connected to a master server at 127.27.185.126:42515
I20260929 20:16:48.849778 526 heartbeater.cc:464] Registering TS with master...
I20260929 20:16:48.850366 526 heartbeater.cc:511] Master 127.27.185.126:42515 requested a full tablet report, sending...
I20260929 20:16:48.851655 32540 ts_manager.cc:194] Registered new tserver with Master: 9f6486ecaabd44aabb2385046a619f64 (127.27.185.67:37285)
I20260929 20:16:48.852437 32540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.67@KRBTEST.COM'} at 127.27.185.67:41847
I20260929 20:16:48.858013 28389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:16:48.874082 395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:48.862523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49740 (local address 127.27.185.126:42515)
0929 20:16:48.862693 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:48.862697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:48.863207 (+ 510us) server_negotiation.cc:407] Connection header received
0929 20:16:48.863347 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:48.863350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:48.863462 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:48.863544 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:48.864940 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:48.865642 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:48.866838 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:48.867242 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:48.867331 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:48.869515 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:48.869520 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:48.869523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:48.869553 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:48.871946 (+ 2393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:48.872506 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:48.872509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:48.872512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:48.872619 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:48.872917 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:48.872920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:48.872933 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:48.873229 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:48.873410 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:48.873661 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 20:16:48.873818 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:16:48.878296 32540 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:16:48.885056 32540 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:16:48.924060 539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:48.897054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:33201 (local address 127.0.0.1:39372)
0929 20:16:48.910396 (+ 13342us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:48.910464 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:48.910747 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:48.916175 (+ 5428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:48.916187 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:48.916230 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:16:48.916745 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:48.916759 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:48.918574 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:48.918580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:48.921183 (+ 2603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:48.921197 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:48.923228 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:48.923234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:48.923442 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:48.923470 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 20:16:48.923546 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 20:16:48.923645 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12941,"mutex_wait_us":24,"thread_start_us":14477,"threads_started":1}
I20260929 20:16:48.924233 543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:48.912320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59328 (local address 127.27.185.65:38505)
0929 20:16:48.916830 (+ 4510us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:48.916841 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:48.916869 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:16:48.916971 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:48.916981 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:48.917209 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:48.917414 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:16:48.917993 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:48.919535 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:48.921792 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:48.923097 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:48.923238 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:48.923507 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:48.923747 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 20:16:48.923904 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4317,"thread_start_us":4136,"threads_started":1}
I20260929 20:16:48.924538 541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:48.897621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41982 (local address 127.27.185.67:37285)
0929 20:16:48.915482 (+ 17861us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:48.915492 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:48.915529 (+ 37us) server_negotiation.cc:407] Connection header received
0929 20:16:48.915637 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:48.915645 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:48.915880 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:48.916085 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:16:48.917227 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:48.918865 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:48.921015 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:48.922413 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:48.922546 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:48.922788 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:48.924057 (+ 1269us) server_negotiation.cc:299] Negotiation successful
0929 20:16:48.924236 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17672,"thread_start_us":17509,"threads_started":1}
I20260929 20:16:48.924211 542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:48.911970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:38505 (local address 127.0.0.1:59328)
0929 20:16:48.914064 (+ 2094us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:48.914087 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:48.914225 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:48.917416 (+ 3191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:48.917420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:48.917445 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:16:48.917781 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:48.917791 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:48.919672 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:48.919678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:48.921649 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:48.921659 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:48.923346 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:48.923352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:48.923562 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:48.923586 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 20:16:48.923688 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 20:16:48.923820 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1923,"thread_start_us":2334,"threads_started":1}
I20260929 20:16:48.924888 538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:48.897318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:37285 (local address 127.0.0.1:41982)
0929 20:16:48.911305 (+ 13987us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:48.911324 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:48.911454 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:48.916371 (+ 4917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:48.916378 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:48.916417 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:16:48.916914 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:48.916927 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:48.919042 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:48.919047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:48.920846 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:48.920858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:48.922945 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:48.922952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:48.923146 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:48.923216 (+ 70us) client_negotiation.cc:769] Sending connection context
0929 20:16:48.923350 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 20:16:48.923573 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13874,"mutex_wait_us":1,"thread_start_us":13720,"threads_started":1}
I20260929 20:16:48.925081 540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:48.897378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39372 (local address 127.27.185.66:33201)
0929 20:16:48.915023 (+ 17645us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:48.915037 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:48.915076 (+ 39us) server_negotiation.cc:407] Connection header received
0929 20:16:48.915222 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:48.915235 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:48.915599 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:48.915845 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:16:48.916921 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:48.918375 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:48.921354 (+ 2979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:48.922765 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:48.922903 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:48.923092 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:48.924373 (+ 1281us) server_negotiation.cc:299] Negotiation successful
0929 20:16:48.924539 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17417,"thread_start_us":17249,"threads_started":1}
I20260929 20:16:48.928521 461 tablet_service.cc:1467] Processing CreateTablet for tablet 542e9f5ab2254acab774092ea49de8f7 (DEFAULT_TABLE table=test-table [id=290e9cb861be4a3293cf47ca90847a51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:16:48.928512 32663 tablet_service.cc:1467] Processing CreateTablet for tablet 542e9f5ab2254acab774092ea49de8f7 (DEFAULT_TABLE table=test-table [id=290e9cb861be4a3293cf47ca90847a51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:16:48.928529 328 tablet_service.cc:1467] Processing CreateTablet for tablet 542e9f5ab2254acab774092ea49de8f7 (DEFAULT_TABLE table=test-table [id=290e9cb861be4a3293cf47ca90847a51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:16:48.929883 328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 542e9f5ab2254acab774092ea49de8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:48.929921 32663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 542e9f5ab2254acab774092ea49de8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:48.929922 461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 542e9f5ab2254acab774092ea49de8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:48.944478 545 tablet_bootstrap.cc:492] T 542e9f5ab2254acab774092ea49de8f7 P 9f6486ecaabd44aabb2385046a619f64: Bootstrap starting.
I20260929 20:16:48.945617 544 tablet_bootstrap.cc:492] T 542e9f5ab2254acab774092ea49de8f7 P f3ffdbf04d034b1e89d34aa52659e3f1: Bootstrap starting.
I20260929 20:16:48.947396 546 tablet_bootstrap.cc:492] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2: Bootstrap starting.
I20260929 20:16:48.948625 545 tablet_bootstrap.cc:654] T 542e9f5ab2254acab774092ea49de8f7 P 9f6486ecaabd44aabb2385046a619f64: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:48.949841 544 tablet_bootstrap.cc:654] T 542e9f5ab2254acab774092ea49de8f7 P f3ffdbf04d034b1e89d34aa52659e3f1: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:48.949949 545 log.cc:824] T 542e9f5ab2254acab774092ea49de8f7 P 9f6486ecaabd44aabb2385046a619f64: Log is configured to *not* fsync() on all Append() calls
I20260929 20:16:48.951001 546 tablet_bootstrap.cc:654] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:48.950994 544 log.cc:824] T 542e9f5ab2254acab774092ea49de8f7 P f3ffdbf04d034b1e89d34aa52659e3f1: Log is configured to *not* fsync() on all Append() calls
I20260929 20:16:48.952224 546 log.cc:824] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2: Log is configured to *not* fsync() on all Append() calls
I20260929 20:16:48.953207 545 tablet_bootstrap.cc:492] T 542e9f5ab2254acab774092ea49de8f7 P 9f6486ecaabd44aabb2385046a619f64: No bootstrap required, opened a new log
I20260929 20:16:48.953501 545 ts_tablet_manager.cc:1397] T 542e9f5ab2254acab774092ea49de8f7 P 9f6486ecaabd44aabb2385046a619f64: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 20:16:48.953816 544 tablet_bootstrap.cc:492] T 542e9f5ab2254acab774092ea49de8f7 P f3ffdbf04d034b1e89d34aa52659e3f1: No bootstrap required, opened a new log
I20260929 20:16:48.954062 544 ts_tablet_manager.cc:1397] T 542e9f5ab2254acab774092ea49de8f7 P f3ffdbf04d034b1e89d34aa52659e3f1: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260929 20:16:48.955175 546 tablet_bootstrap.cc:492] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2: No bootstrap required, opened a new log
I20260929 20:16:48.955438 546 ts_tablet_manager.cc:1397] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 20:16:48.961884 544 raft_consensus.cc:348] T 542e9f5ab2254acab774092ea49de8f7 P f3ffdbf04d034b1e89d34aa52659e3f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ffdbf04d034b1e89d34aa52659e3f1" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33201 } } peers { permanent_uuid: "9f6486ecaabd44aabb2385046a619f64" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 37285 } } peers { permanent_uuid: "eeb9937f42f64596b59fe0ea25b42da2" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38505 } }
I20260929 20:16:48.962270 544 raft_consensus.cc:374] T 542e9f5ab2254acab774092ea49de8f7 P f3ffdbf04d034b1e89d34aa52659e3f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:48.962370 544 raft_consensus.cc:729] T 542e9f5ab2254acab774092ea49de8f7 P f3ffdbf04d034b1e89d34aa52659e3f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3ffdbf04d034b1e89d34aa52659e3f1, State: Initialized, Role: FOLLOWER
I20260929 20:16:48.963019 544 consensus_queue.cc:260] T 542e9f5ab2254acab774092ea49de8f7 P f3ffdbf04d034b1e89d34aa52659e3f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ffdbf04d034b1e89d34aa52659e3f1" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33201 } } peers { permanent_uuid: "9f6486ecaabd44aabb2385046a619f64" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 37285 } } peers { permanent_uuid: "eeb9937f42f64596b59fe0ea25b42da2" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38505 } }
I20260929 20:16:48.964210 393 heartbeater.cc:503] Master 127.27.185.126:42515 was elected leader, sending a full tablet report...
I20260929 20:16:48.964867 544 ts_tablet_manager.cc:1428] T 542e9f5ab2254acab774092ea49de8f7 P f3ffdbf04d034b1e89d34aa52659e3f1: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260929 20:16:48.965777 545 raft_consensus.cc:348] T 542e9f5ab2254acab774092ea49de8f7 P 9f6486ecaabd44aabb2385046a619f64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ffdbf04d034b1e89d34aa52659e3f1" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33201 } } peers { permanent_uuid: "9f6486ecaabd44aabb2385046a619f64" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 37285 } } peers { permanent_uuid: "eeb9937f42f64596b59fe0ea25b42da2" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38505 } }
I20260929 20:16:48.965777 546 raft_consensus.cc:348] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ffdbf04d034b1e89d34aa52659e3f1" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33201 } } peers { permanent_uuid: "9f6486ecaabd44aabb2385046a619f64" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 37285 } } peers { permanent_uuid: "eeb9937f42f64596b59fe0ea25b42da2" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38505 } }
I20260929 20:16:48.966219 546 raft_consensus.cc:374] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:48.966219 545 raft_consensus.cc:374] T 542e9f5ab2254acab774092ea49de8f7 P 9f6486ecaabd44aabb2385046a619f64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:48.966331 545 raft_consensus.cc:729] T 542e9f5ab2254acab774092ea49de8f7 P 9f6486ecaabd44aabb2385046a619f64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6486ecaabd44aabb2385046a619f64, State: Initialized, Role: FOLLOWER
I20260929 20:16:48.966321 546 raft_consensus.cc:729] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb9937f42f64596b59fe0ea25b42da2, State: Initialized, Role: FOLLOWER
I20260929 20:16:48.966945 546 consensus_queue.cc:260] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ffdbf04d034b1e89d34aa52659e3f1" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33201 } } peers { permanent_uuid: "9f6486ecaabd44aabb2385046a619f64" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 37285 } } peers { permanent_uuid: "eeb9937f42f64596b59fe0ea25b42da2" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38505 } }
I20260929 20:16:48.966934 545 consensus_queue.cc:260] T 542e9f5ab2254acab774092ea49de8f7 P 9f6486ecaabd44aabb2385046a619f64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ffdbf04d034b1e89d34aa52659e3f1" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33201 } } peers { permanent_uuid: "9f6486ecaabd44aabb2385046a619f64" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 37285 } } peers { permanent_uuid: "eeb9937f42f64596b59fe0ea25b42da2" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38505 } }
I20260929 20:16:48.968689 526 heartbeater.cc:503] Master 127.27.185.126:42515 was elected leader, sending a full tablet report...
I20260929 20:16:48.968833 546 ts_tablet_manager.cc:1428] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260929 20:16:48.971539 545 ts_tablet_manager.cc:1428] T 542e9f5ab2254acab774092ea49de8f7 P 9f6486ecaabd44aabb2385046a619f64: Time spent starting tablet: real 0.018s user 0.013s sys 0.003s
W20260929 20:16:49.054663 32729 tablet.cc:2369] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:16:49.078694 527 tablet.cc:2369] T 542e9f5ab2254acab774092ea49de8f7 P 9f6486ecaabd44aabb2385046a619f64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:16:49.079080 552 raft_consensus.cc:482] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:16:49.079419 552 raft_consensus.cc:504] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ffdbf04d034b1e89d34aa52659e3f1" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33201 } } peers { permanent_uuid: "9f6486ecaabd44aabb2385046a619f64" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 37285 } } peers { permanent_uuid: "eeb9937f42f64596b59fe0ea25b42da2" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38505 } }
I20260929 20:16:49.081169 552 leader_election.cc:288] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3ffdbf04d034b1e89d34aa52659e3f1 (127.27.185.66:33201), 9f6486ecaabd44aabb2385046a619f64 (127.27.185.67:37285)
W20260929 20:16:49.084331 394 tablet.cc:2369] T 542e9f5ab2254acab774092ea49de8f7 P f3ffdbf04d034b1e89d34aa52659e3f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:16:49.087743 541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:49.081569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:42727 (local address 127.27.185.67:37285)
0929 20:16:49.081786 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:49.081794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:49.082275 (+ 481us) server_negotiation.cc:407] Connection header received
0929 20:16:49.082518 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:49.082528 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:49.082591 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:49.082687 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:16:49.083267 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:49.084043 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:49.086007 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:49.086781 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:49.086976 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:49.087051 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:49.087324 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 20:16:49.087490 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:16:49.087867 553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:49.081439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:37285 (local address 127.27.185.65:42727)
0929 20:16:49.082165 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:49.082182 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:49.082364 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:49.082708 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:49.082713 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:49.082800 (+ 87us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:16:49.083122 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:49.083130 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:49.084204 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:49.084209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:49.085843 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:49.085855 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:49.086999 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:49.087005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:49.087165 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:49.087188 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:16:49.087299 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 20:16:49.087452 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":23,"spinlock_wait_cycles":24832,"thread_start_us":320,"threads_started":1}
I20260929 20:16:49.088111 540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:49.081619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:56575 (local address 127.27.185.66:33201)
0929 20:16:49.081849 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:49.081857 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:49.082689 (+ 832us) server_negotiation.cc:407] Connection header received
0929 20:16:49.082894 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:49.082898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:49.082957 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:49.083055 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:16:49.083647 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:49.084492 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:49.086400 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:49.087149 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:49.087333 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:49.087425 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:49.087713 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 20:16:49.087844 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 20:16:49.088589 554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:49.081439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:33201 (local address 127.27.185.65:56575)
0929 20:16:49.082576 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:49.082590 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:49.082739 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:49.083075 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:49.083083 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:49.083109 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:16:49.083494 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:49.083504 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:49.084649 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:49.084655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:49.086250 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:49.086262 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:49.087363 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:49.087369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:49.087556 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:49.087582 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 20:16:49.087687 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 20:16:49.088098 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"mutex_wait_us":450,"thread_start_us":408,"threads_started":1}
I20260929 20:16:49.088902 481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "542e9f5ab2254acab774092ea49de8f7" candidate_uuid: "eeb9937f42f64596b59fe0ea25b42da2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6486ecaabd44aabb2385046a619f64" is_pre_election: true
I20260929 20:16:49.089522 481 raft_consensus.cc:2454] T 542e9f5ab2254acab774092ea49de8f7 P 9f6486ecaabd44aabb2385046a619f64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeb9937f42f64596b59fe0ea25b42da2 in term 0.
I20260929 20:16:49.089553 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "542e9f5ab2254acab774092ea49de8f7" candidate_uuid: "eeb9937f42f64596b59fe0ea25b42da2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3ffdbf04d034b1e89d34aa52659e3f1" is_pre_election: true
I20260929 20:16:49.090104 348 raft_consensus.cc:2454] T 542e9f5ab2254acab774092ea49de8f7 P f3ffdbf04d034b1e89d34aa52659e3f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeb9937f42f64596b59fe0ea25b42da2 in term 0.
I20260929 20:16:49.090169 32618 leader_election.cc:302] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f6486ecaabd44aabb2385046a619f64, eeb9937f42f64596b59fe0ea25b42da2; no voters:
I20260929 20:16:49.090679 552 raft_consensus.cc:2781] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:16:49.090870 552 raft_consensus.cc:482] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:16:49.091001 552 raft_consensus.cc:3037] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:49.093284 552 raft_consensus.cc:504] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ffdbf04d034b1e89d34aa52659e3f1" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33201 } } peers { permanent_uuid: "9f6486ecaabd44aabb2385046a619f64" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 37285 } } peers { permanent_uuid: "eeb9937f42f64596b59fe0ea25b42da2" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38505 } }
I20260929 20:16:49.093926 552 leader_election.cc:288] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2 [CANDIDATE]: Term 1 election: Requested vote from peers f3ffdbf04d034b1e89d34aa52659e3f1 (127.27.185.66:33201), 9f6486ecaabd44aabb2385046a619f64 (127.27.185.67:37285)
I20260929 20:16:49.094408 481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "542e9f5ab2254acab774092ea49de8f7" candidate_uuid: "eeb9937f42f64596b59fe0ea25b42da2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6486ecaabd44aabb2385046a619f64"
I20260929 20:16:49.094275 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "542e9f5ab2254acab774092ea49de8f7" candidate_uuid: "eeb9937f42f64596b59fe0ea25b42da2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3ffdbf04d034b1e89d34aa52659e3f1"
I20260929 20:16:49.094679 348 raft_consensus.cc:3037] T 542e9f5ab2254acab774092ea49de8f7 P f3ffdbf04d034b1e89d34aa52659e3f1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:49.094585 481 raft_consensus.cc:3037] T 542e9f5ab2254acab774092ea49de8f7 P 9f6486ecaabd44aabb2385046a619f64 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:49.097041 348 raft_consensus.cc:2454] T 542e9f5ab2254acab774092ea49de8f7 P f3ffdbf04d034b1e89d34aa52659e3f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeb9937f42f64596b59fe0ea25b42da2 in term 1.
I20260929 20:16:49.097045 481 raft_consensus.cc:2454] T 542e9f5ab2254acab774092ea49de8f7 P 9f6486ecaabd44aabb2385046a619f64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeb9937f42f64596b59fe0ea25b42da2 in term 1.
I20260929 20:16:49.097496 32619 leader_election.cc:302] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eeb9937f42f64596b59fe0ea25b42da2, f3ffdbf04d034b1e89d34aa52659e3f1; no voters:
I20260929 20:16:49.097805 552 raft_consensus.cc:2781] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:16:49.098237 552 raft_consensus.cc:686] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2 [term 1 LEADER]: Becoming Leader. State: Replica: eeb9937f42f64596b59fe0ea25b42da2, State: Running, Role: LEADER
I20260929 20:16:49.098804 552 consensus_queue.cc:237] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ffdbf04d034b1e89d34aa52659e3f1" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33201 } } peers { permanent_uuid: "9f6486ecaabd44aabb2385046a619f64" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 37285 } } peers { permanent_uuid: "eeb9937f42f64596b59fe0ea25b42da2" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38505 } }
I20260929 20:16:49.103308 32539 catalog_manager.cc:5495] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2 reported cstate change: term changed from 0 to 1, leader changed from <none> to eeb9937f42f64596b59fe0ea25b42da2 (127.27.185.65). New cstate: current_term: 1 leader_uuid: "eeb9937f42f64596b59fe0ea25b42da2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ffdbf04d034b1e89d34aa52659e3f1" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f6486ecaabd44aabb2385046a619f64" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 37285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeb9937f42f64596b59fe0ea25b42da2" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 38505 } health_report { overall_health: HEALTHY } } }
I20260929 20:16:49.128430 543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:49.123536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59332 (local address 127.27.185.65:38505)
0929 20:16:49.123754 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:49.123759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:49.123776 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:16:49.123887 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:49.123891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:49.123964 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:49.124093 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:16:49.124425 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:49.125128 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:49.126386 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:49.126768 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:49.126879 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:49.127305 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:16:49.127870 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:16:49.127988 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:49.128069 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 20:16:49.128134 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:16:49.136622 348 raft_consensus.cc:1261] T 542e9f5ab2254acab774092ea49de8f7 P f3ffdbf04d034b1e89d34aa52659e3f1 [term 1 FOLLOWER]: Refusing update from remote peer eeb9937f42f64596b59fe0ea25b42da2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:16:49.136690 481 raft_consensus.cc:1261] T 542e9f5ab2254acab774092ea49de8f7 P 9f6486ecaabd44aabb2385046a619f64 [term 1 FOLLOWER]: Refusing update from remote peer eeb9937f42f64596b59fe0ea25b42da2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:16:49.137660 555 consensus_queue.cc:1035] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3ffdbf04d034b1e89d34aa52659e3f1" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:16:49.138111 552 consensus_queue.cc:1035] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f6486ecaabd44aabb2385046a619f64" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 37285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:16:49.150141 558 mvcc.cc:205] Tried to move back new op lower bound from 7334760485413412864 to 7334760485272735744. Current Snapshot: MvccSnapshot[applied={T|T < 7334760485413412864}]
I20260929 20:16:49.163908 32540 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:16:49.164209 32540 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:16:49.167583 32540 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c3cbc69eac04aeea8a4acb70e8412ee: Sending DeleteTablet for 3 replicas of tablet 542e9f5ab2254acab774092ea49de8f7
I20260929 20:16:49.169037 32663 tablet_service.cc:1514] Processing DeleteTablet for tablet 542e9f5ab2254acab774092ea49de8f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:16:49 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:59328
I20260929 20:16:49.169034 328 tablet_service.cc:1514] Processing DeleteTablet for tablet 542e9f5ab2254acab774092ea49de8f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:16:49 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:39372
I20260929 20:16:49.169163 461 tablet_service.cc:1514] Processing DeleteTablet for tablet 542e9f5ab2254acab774092ea49de8f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:16:49 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:41982
I20260929 20:16:49.171233 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 32601
I20260929 20:16:49.179992 570 tablet_replica.cc:330] T 542e9f5ab2254acab774092ea49de8f7 P 9f6486ecaabd44aabb2385046a619f64: stopping tablet replica
I20260929 20:16:49.180572 570 raft_consensus.cc:2229] T 542e9f5ab2254acab774092ea49de8f7 P 9f6486ecaabd44aabb2385046a619f64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:16:49.182046 569 tablet_replica.cc:330] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2: stopping tablet replica
I20260929 20:16:49.182565 569 raft_consensus.cc:2229] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:16:49.183481 569 raft_consensus.cc:2258] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:16:49.183908 570 raft_consensus.cc:2258] T 542e9f5ab2254acab774092ea49de8f7 P 9f6486ecaabd44aabb2385046a619f64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:16:49.184525 568 tablet_replica.cc:330] T 542e9f5ab2254acab774092ea49de8f7 P f3ffdbf04d034b1e89d34aa52659e3f1: stopping tablet replica
I20260929 20:16:49.184930 568 raft_consensus.cc:2229] T 542e9f5ab2254acab774092ea49de8f7 P f3ffdbf04d034b1e89d34aa52659e3f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:16:49.185439 568 raft_consensus.cc:2258] T 542e9f5ab2254acab774092ea49de8f7 P f3ffdbf04d034b1e89d34aa52659e3f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:16:49.186317 570 ts_tablet_manager.cc:1905] T 542e9f5ab2254acab774092ea49de8f7 P 9f6486ecaabd44aabb2385046a619f64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:16:49.187443 568 ts_tablet_manager.cc:1905] T 542e9f5ab2254acab774092ea49de8f7 P f3ffdbf04d034b1e89d34aa52659e3f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:16:49.188696 569 ts_tablet_manager.cc:1905] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:16:49.193383 570 ts_tablet_manager.cc:1918] T 542e9f5ab2254acab774092ea49de8f7 P 9f6486ecaabd44aabb2385046a619f64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:16:49.193560 570 log.cc:1196] T 542e9f5ab2254acab774092ea49de8f7 P 9f6486ecaabd44aabb2385046a619f64: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-2/wal/wals/542e9f5ab2254acab774092ea49de8f7
I20260929 20:16:49.194059 570 ts_tablet_manager.cc:1939] T 542e9f5ab2254acab774092ea49de8f7 P 9f6486ecaabd44aabb2385046a619f64: Deleting consensus metadata
I20260929 20:16:49.196213 32526 catalog_manager.cc:4841] TS 9f6486ecaabd44aabb2385046a619f64 (127.27.185.67:37285): tablet 542e9f5ab2254acab774092ea49de8f7 (table test-table [id=290e9cb861be4a3293cf47ca90847a51]) successfully deleted
I20260929 20:16:49.197784 568 ts_tablet_manager.cc:1918] T 542e9f5ab2254acab774092ea49de8f7 P f3ffdbf04d034b1e89d34aa52659e3f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Sep 29 20:16:49 dist-test-slave-4cx3 krb5kdc[32496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.65@KRBTEST.COM
I20260929 20:16:49.197953 568 log.cc:1196] T 542e9f5ab2254acab774092ea49de8f7 P f3ffdbf04d034b1e89d34aa52659e3f1: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-1/wal/wals/542e9f5ab2254acab774092ea49de8f7
I20260929 20:16:49.198437 568 ts_tablet_manager.cc:1939] T 542e9f5ab2254acab774092ea49de8f7 P f3ffdbf04d034b1e89d34aa52659e3f1: Deleting consensus metadata
I20260929 20:16:49.200467 32527 catalog_manager.cc:4841] TS f3ffdbf04d034b1e89d34aa52659e3f1 (127.27.185.66:33201): tablet 542e9f5ab2254acab774092ea49de8f7 (table test-table [id=290e9cb861be4a3293cf47ca90847a51]) successfully deleted
I20260929 20:16:49.203802 569 ts_tablet_manager.cc:1918] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:16:49.203974 569 log.cc:1196] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790712966917858-28389-0/minicluster-data/ts-0/wal/wals/542e9f5ab2254acab774092ea49de8f7
I20260929 20:16:49.204465 569 ts_tablet_manager.cc:1939] T 542e9f5ab2254acab774092ea49de8f7 P eeb9937f42f64596b59fe0ea25b42da2: Deleting consensus metadata
I20260929 20:16:49.206108 32526 catalog_manager.cc:4841] TS eeb9937f42f64596b59fe0ea25b42da2 (127.27.185.65:38505): tablet 542e9f5ab2254acab774092ea49de8f7 (table test-table [id=290e9cb861be4a3293cf47ca90847a51]) successfully deleted
I20260929 20:16:49.207405 543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:49.174798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59344 (local address 127.27.185.65:38505)
0929 20:16:49.175036 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:49.175045 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:49.187028 (+ 11983us) server_negotiation.cc:407] Connection header received
0929 20:16:49.187254 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:49.187260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:49.187342 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:49.187459 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:49.189491 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:49.190624 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:49.192605 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:49.193321 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:49.193429 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:49.199255 (+ 5826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:49.199271 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:49.199295 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:49.199345 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:49.203500 (+ 4155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:49.204664 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:49.204676 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:49.204685 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:49.204871 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:49.205327 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:49.205333 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:49.205337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:49.205816 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:49.206093 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:49.206752 (+ 659us) server_negotiation.cc:299] Negotiation successful
0929 20:16:49.207015 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:16:49.208987 32723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:16:49.305364 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 32601
I20260929 20:16:49.320739 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 32734
Sep 29 20:16:49 dist-test-slave-4cx3 krb5kdc[32496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.66@KRBTEST.COM
I20260929 20:16:49.337230 540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:49.321796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39386 (local address 127.27.185.66:33201)
0929 20:16:49.321998 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:49.322005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:49.322026 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:16:49.322102 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:49.322106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:49.322173 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:49.322313 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:49.323597 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:49.324392 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:49.326173 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:49.326599 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:49.326698 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:49.331091 (+ 4393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:49.331104 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:49.331119 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:49.331160 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:49.334243 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:49.334944 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:49.334952 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:49.334955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:49.335086 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:49.335498 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:49.335504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:49.335508 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:49.336008 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:49.336296 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:49.336609 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 20:16:49.336840 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:16:49.337986 388 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:16:49.427114 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 32734
I20260929 20:16:49.443915 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 399
Sep 29 20:16:49 dist-test-slave-4cx3 krb5kdc[32496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.67@KRBTEST.COM
I20260929 20:16:49.458590 541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:49.444681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41988 (local address 127.27.185.67:37285)
0929 20:16:49.444922 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:49.444927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:49.444946 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:16:49.445073 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:49.445077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:49.445143 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:49.445229 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:49.446447 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:49.447170 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:49.448377 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:49.448754 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:49.448859 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:49.453056 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:49.453073 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:49.453090 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:49.453127 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:49.456039 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:49.456603 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:49.456610 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:49.456613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:49.456815 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:49.457135 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:49.457138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:49.457140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:49.457506 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:49.457726 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:49.458049 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 20:16:49.458278 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:16:49.459226 521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:16:49.548061 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 399
I20260929 20:16:49.564337 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 32509
I20260929 20:16:49.565510 32570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:16:49.675364 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 32509
2026-09-29T20:16:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4247 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T20:16:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:16:49Z Disabled control of system clock
I20260929 20:16:49.743077 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:41631
--webserver_interface=127.27.185.126
--webserver_port=0
--builtin_ntp_servers=127.27.185.84:43953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:41631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 20:16:50.065526 579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:50.065838 579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:50.065977 579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:50.073544 579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:16:50.073673 579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:50.073755 579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:50.073864 579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:16:50.073928 579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:16:50.083585 579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:43953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:41631
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:41631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:50.085057 579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:50.086973 579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:50.099428 584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:50.099308 587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:50.099398 585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:50.099733 579 server_base.cc:1034] running on GCE node
I20260929 20:16:50.100749 579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:50.102213 579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:50.103520 579 hybrid_clock.cc:648] HybridClock initialized: now 1790713010103463 us; error 53 us; skew 500 ppm
I20260929 20:16:50.103873 579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:16:50.110466 579 webserver.cc:460] Webserver started at http://127.27.185.126:39023/ using document root <none> and password file <none>
I20260929 20:16:50.111243 579 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:50.111418 579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:50.111765 579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:50.114745 579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "0cc7a445f9ea40db97ca0ab0ab9e6f6a"
format_stamp: "Formatted at 2026-09-29 20:16:50 on dist-test-slave-4cx3"
server_key: "92845bf3f048a98307a6b9cb0f27add5"
server_key_iv: "fe0cd44015c78c1c9e516de287087387"
server_key_version: "encryptionkey@0"
I20260929 20:16:50.115622 579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "0cc7a445f9ea40db97ca0ab0ab9e6f6a"
format_stamp: "Formatted at 2026-09-29 20:16:50 on dist-test-slave-4cx3"
server_key: "92845bf3f048a98307a6b9cb0f27add5"
server_key_iv: "fe0cd44015c78c1c9e516de287087387"
server_key_version: "encryptionkey@0"
I20260929 20:16:50.120780 579 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 20:16:50.124284 593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:50.125635 579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 20:16:50.125833 579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "0cc7a445f9ea40db97ca0ab0ab9e6f6a"
format_stamp: "Formatted at 2026-09-29 20:16:50 on dist-test-slave-4cx3"
server_key: "92845bf3f048a98307a6b9cb0f27add5"
server_key_iv: "fe0cd44015c78c1c9e516de287087387"
server_key_version: "encryptionkey@0"
I20260929 20:16:50.126062 579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:50.131965 579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:50.132762 579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:50.133062 579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:50.150866 579 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:41631
I20260929 20:16:50.150877 644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:41631 every 8 connection(s)
I20260929 20:16:50.152220 579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:16:50.155861 645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:50.161264 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 579
I20260929 20:16:50.161468 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:16:50.162106 28389 external_mini_cluster.cc:1428] Setting key b8ae71d9da6283a92d8c93e1250d87ff
I20260929 20:16:50.166559 645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cc7a445f9ea40db97ca0ab0ab9e6f6a: Bootstrap starting.
I20260929 20:16:50.170264 645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cc7a445f9ea40db97ca0ab0ab9e6f6a: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:50.170639 648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:50.164327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49240 (local address 127.27.185.126:41631)
0929 20:16:50.165326 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:50.165337 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:50.165377 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:16:50.167054 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:50.167077 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:50.167553 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:50.167861 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:50.169017 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:50.169023 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:50.169036 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:16:50.169049 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:50.169111 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:16:50.169127 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:50.169300 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:50.169435 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 20:16:50.169707 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":368,"threads_started":1}
I20260929 20:16:50.171658 645 log.cc:824] T 00000000000000000000000000000000 P 0cc7a445f9ea40db97ca0ab0ab9e6f6a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:16:50.174651 645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cc7a445f9ea40db97ca0ab0ab9e6f6a: No bootstrap required, opened a new log
I20260929 20:16:50.181897 645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cc7a445f9ea40db97ca0ab0ab9e6f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc7a445f9ea40db97ca0ab0ab9e6f6a" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41631 } }
I20260929 20:16:50.182255 645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cc7a445f9ea40db97ca0ab0ab9e6f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:50.182327 645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cc7a445f9ea40db97ca0ab0ab9e6f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cc7a445f9ea40db97ca0ab0ab9e6f6a, State: Initialized, Role: FOLLOWER
I20260929 20:16:50.182847 645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cc7a445f9ea40db97ca0ab0ab9e6f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc7a445f9ea40db97ca0ab0ab9e6f6a" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41631 } }
I20260929 20:16:50.183051 645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cc7a445f9ea40db97ca0ab0ab9e6f6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:16:50.183149 645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cc7a445f9ea40db97ca0ab0ab9e6f6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:16:50.183323 645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cc7a445f9ea40db97ca0ab0ab9e6f6a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:50.185765 645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cc7a445f9ea40db97ca0ab0ab9e6f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc7a445f9ea40db97ca0ab0ab9e6f6a" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41631 } }
I20260929 20:16:50.186275 645 leader_election.cc:302] T 00000000000000000000000000000000 P 0cc7a445f9ea40db97ca0ab0ab9e6f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cc7a445f9ea40db97ca0ab0ab9e6f6a; no voters:
I20260929 20:16:50.186846 645 leader_election.cc:288] T 00000000000000000000000000000000 P 0cc7a445f9ea40db97ca0ab0ab9e6f6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:16:50.186988 650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cc7a445f9ea40db97ca0ab0ab9e6f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:16:50.187788 650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cc7a445f9ea40db97ca0ab0ab9e6f6a [term 1 LEADER]: Becoming Leader. State: Replica: 0cc7a445f9ea40db97ca0ab0ab9e6f6a, State: Running, Role: LEADER
I20260929 20:16:50.188418 650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cc7a445f9ea40db97ca0ab0ab9e6f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc7a445f9ea40db97ca0ab0ab9e6f6a" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41631 } }
I20260929 20:16:50.189188 645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cc7a445f9ea40db97ca0ab0ab9e6f6a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:16:50.191558 651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cc7a445f9ea40db97ca0ab0ab9e6f6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cc7a445f9ea40db97ca0ab0ab9e6f6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc7a445f9ea40db97ca0ab0ab9e6f6a" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41631 } } }
I20260929 20:16:50.191939 651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cc7a445f9ea40db97ca0ab0ab9e6f6a [sys.catalog]: This master's current role is: LEADER
I20260929 20:16:50.192668 652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cc7a445f9ea40db97ca0ab0ab9e6f6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cc7a445f9ea40db97ca0ab0ab9e6f6a. Latest consensus state: current_term: 1 leader_uuid: "0cc7a445f9ea40db97ca0ab0ab9e6f6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc7a445f9ea40db97ca0ab0ab9e6f6a" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41631 } } }
I20260929 20:16:50.192957 652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cc7a445f9ea40db97ca0ab0ab9e6f6a [sys.catalog]: This master's current role is: LEADER
I20260929 20:16:50.192996 657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:16:50.199849 657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:16:50.209451 657 catalog_manager.cc:1347] Generated new cluster ID: bd31d03d724149c0a2f0b98de7e65666
I20260929 20:16:50.209579 657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:16:50.234983 657 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:16:50.235534 657 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:16:50.248009 657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cc7a445f9ea40db97ca0ab0ab9e6f6a: Generated new TSK 0
I20260929 20:16:50.249181 657 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:16:50.258774 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:0
--local_ip_for_outbound_sockets=127.27.185.65
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:41631
--builtin_ntp_servers=127.27.185.84:43953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 20:16:50.590077 669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:50.590453 669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:50.590622 669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:50.597987 669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:50.598116 669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:50.598295 669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:16:50.607215 669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:43953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:41631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:50.608551 669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:50.610330 669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:50.623485 677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:50.623880 675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:50.623894 674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:50.625020 669 server_base.cc:1034] running on GCE node
I20260929 20:16:50.625603 669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:50.626518 669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:50.627709 669 hybrid_clock.cc:648] HybridClock initialized: now 1790713010627670 us; error 26 us; skew 500 ppm
I20260929 20:16:50.628046 669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:16:50.630242 669 webserver.cc:460] Webserver started at http://127.27.185.65:42487/ using document root <none> and password file <none>
I20260929 20:16:50.630999 669 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:50.631116 669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:50.631556 669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:50.635329 669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-0/data/instance:
uuid: "dee81202602c425cb7c97ec67e5250f8"
format_stamp: "Formatted at 2026-09-29 20:16:50 on dist-test-slave-4cx3"
server_key: "bcdf458a9bd81200ca41b5c3a5148739"
server_key_iv: "d0bf58ee2f4824a2426603585e7a5d6e"
server_key_version: "encryptionkey@0"
I20260929 20:16:50.636348 669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance:
uuid: "dee81202602c425cb7c97ec67e5250f8"
format_stamp: "Formatted at 2026-09-29 20:16:50 on dist-test-slave-4cx3"
server_key: "bcdf458a9bd81200ca41b5c3a5148739"
server_key_iv: "d0bf58ee2f4824a2426603585e7a5d6e"
server_key_version: "encryptionkey@0"
I20260929 20:16:50.641486 669 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:16:50.644969 683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:50.646334 669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:16:50.646507 669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "dee81202602c425cb7c97ec67e5250f8"
format_stamp: "Formatted at 2026-09-29 20:16:50 on dist-test-slave-4cx3"
server_key: "bcdf458a9bd81200ca41b5c3a5148739"
server_key_iv: "d0bf58ee2f4824a2426603585e7a5d6e"
server_key_version: "encryptionkey@0"
I20260929 20:16:50.646740 669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:50.652715 669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:50.653522 669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:50.653789 669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:50.654693 669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:16:50.656106 669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:16:50.656216 669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:50.656355 669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:16:50.656466 669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:50.688930 669 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:34843
I20260929 20:16:50.689073 795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:34843 every 8 connection(s)
I20260929 20:16:50.690335 669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:16:50.698869 798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:50.692825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:36347 (local address 127.27.185.126:41631)
0929 20:16:50.693257 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:50.693262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:50.694763 (+ 1501us) server_negotiation.cc:407] Connection header received
0929 20:16:50.696575 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:50.696581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:50.696655 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:50.696772 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:50.698119 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:50.698125 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:50.698130 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:16:50.698150 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:50.698200 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:16:50.698213 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:50.698355 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:50.698528 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 20:16:50.698597 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":228,"threads_started":1}
I20260929 20:16:50.699573 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 669
I20260929 20:16:50.699790 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance
I20260929 20:16:50.700121 799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:50.693460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41631 (local address 127.27.185.65:36347)
0929 20:16:50.694590 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:50.694638 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:50.696292 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:50.697034 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:50.697042 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:50.697134 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:50.697812 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:16:50.697814 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:50.697858 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:16:50.697944 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:50.698305 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:50.698315 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:50.698324 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 20:16:50.698607 (+ 283us) client_negotiation.cc:240] Negotiation successful
0929 20:16:50.698868 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":467,"threads_started":1}
I20260929 20:16:50.700579 28389 external_mini_cluster.cc:1428] Setting key 96f56fa0b1f2382ae06b9fe98f3ead13
I20260929 20:16:50.701853 796 heartbeater.cc:347] Connected to a master server at 127.27.185.126:41631
I20260929 20:16:50.702227 796 heartbeater.cc:464] Registering TS with master...
I20260929 20:16:50.702976 796 heartbeater.cc:511] Master 127.27.185.126:41631 requested a full tablet report, sending...
I20260929 20:16:50.703698 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:0
--local_ip_for_outbound_sockets=127.27.185.66
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:41631
--builtin_ntp_servers=127.27.185.84:43953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 20:16:50.705370 610 ts_manager.cc:194] Registered new tserver with Master: dee81202602c425cb7c97ec67e5250f8 (127.27.185.65:34843)
W20260929 20:16:51.021628 800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:51.021948 800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:51.022089 800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:51.029701 800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:51.029853 800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:51.030027 800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:16:51.039332 800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:43953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:41631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:51.040623 800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:51.042369 800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:51.055290 806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:51.055410 805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:51.055330 808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:51.056034 800 server_base.cc:1034] running on GCE node
I20260929 20:16:51.056663 800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:51.057607 800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:51.058802 800 hybrid_clock.cc:648] HybridClock initialized: now 1790713011058750 us; error 40 us; skew 500 ppm
I20260929 20:16:51.059126 800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:16:51.061419 800 webserver.cc:460] Webserver started at http://127.27.185.66:34417/ using document root <none> and password file <none>
I20260929 20:16:51.062103 800 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:51.062211 800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:51.062551 800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:51.065361 800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-1/data/instance:
uuid: "31401d46097d4e2ab597c4ef0e32f9d1"
format_stamp: "Formatted at 2026-09-29 20:16:51 on dist-test-slave-4cx3"
server_key: "c8409d1860ccbeb3bd43873e193460af"
server_key_iv: "0fd43f7474edceaa9c71e70be7250a73"
server_key_version: "encryptionkey@0"
I20260929 20:16:51.066183 800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance:
uuid: "31401d46097d4e2ab597c4ef0e32f9d1"
format_stamp: "Formatted at 2026-09-29 20:16:51 on dist-test-slave-4cx3"
server_key: "c8409d1860ccbeb3bd43873e193460af"
server_key_iv: "0fd43f7474edceaa9c71e70be7250a73"
server_key_version: "encryptionkey@0"
I20260929 20:16:51.071274 800 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:16:51.074718 814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:51.076078 800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:16:51.076256 800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "31401d46097d4e2ab597c4ef0e32f9d1"
format_stamp: "Formatted at 2026-09-29 20:16:51 on dist-test-slave-4cx3"
server_key: "c8409d1860ccbeb3bd43873e193460af"
server_key_iv: "0fd43f7474edceaa9c71e70be7250a73"
server_key_version: "encryptionkey@0"
I20260929 20:16:51.076494 800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:51.081965 800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:51.082767 800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:51.083034 800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:51.084028 800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:16:51.085453 800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:16:51.085579 800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:51.085716 800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:16:51.085803 800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:51.118031 800 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:39667
I20260929 20:16:51.118067 926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:39667 every 8 connection(s)
I20260929 20:16:51.119330 800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:16:51.123205 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 800
I20260929 20:16:51.123413 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance
I20260929 20:16:51.123986 28389 external_mini_cluster.cc:1428] Setting key e26ab7324ae694999769ad14331e4a85
I20260929 20:16:51.127061 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:0
--local_ip_for_outbound_sockets=127.27.185.67
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:41631
--builtin_ntp_servers=127.27.185.84:43953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 20:16:51.128172 798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:51.121877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:34079 (local address 127.27.185.126:41631)
0929 20:16:51.122058 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:51.122062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:51.123574 (+ 1512us) server_negotiation.cc:407] Connection header received
0929 20:16:51.125689 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:51.125695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:51.125755 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:51.125865 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:51.127341 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:51.127349 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:51.127356 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:16:51.127379 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:51.127451 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:16:51.127469 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:51.127572 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:51.127902 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 20:16:51.127954 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:16:51.129515 929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:51.122335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41631 (local address 127.27.185.66:34079)
0929 20:16:51.123372 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:51.123456 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:51.125398 (+ 1942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:51.126133 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:51.126144 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:51.126264 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:51.126959 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:16:51.126962 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:51.127016 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:16:51.127137 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:51.127669 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:51.127681 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:51.127691 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 20:16:51.127968 (+ 277us) client_negotiation.cc:240] Negotiation successful
0929 20:16:51.128228 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":261,"threads_started":1}
I20260929 20:16:51.130883 927 heartbeater.cc:347] Connected to a master server at 127.27.185.126:41631
I20260929 20:16:51.131188 927 heartbeater.cc:464] Registering TS with master...
I20260929 20:16:51.131889 927 heartbeater.cc:511] Master 127.27.185.126:41631 requested a full tablet report, sending...
I20260929 20:16:51.133399 610 ts_manager.cc:194] Registered new tserver with Master: 31401d46097d4e2ab597c4ef0e32f9d1 (127.27.185.66:39667)
W20260929 20:16:51.443996 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:51.444270 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:51.444438 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:51.451855 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:51.451957 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:51.452127 930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:16:51.461191 930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:43953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:41631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:51.462502 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:51.464318 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:51.477332 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:51.477444 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:51.477458 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:51.478361 930 server_base.cc:1034] running on GCE node
I20260929 20:16:51.478901 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:51.479919 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:51.481163 930 hybrid_clock.cc:648] HybridClock initialized: now 1790713011481103 us; error 49 us; skew 500 ppm
I20260929 20:16:51.481501 930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:16:51.483644 930 webserver.cc:460] Webserver started at http://127.27.185.67:43633/ using document root <none> and password file <none>
I20260929 20:16:51.484308 930 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:51.484442 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:51.484768 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:51.487553 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-2/data/instance:
uuid: "cd78e56b0d864cbe83dd1dbd8f239f3b"
format_stamp: "Formatted at 2026-09-29 20:16:51 on dist-test-slave-4cx3"
server_key: "ad8e5b177c2f0084c754c4b0f9c5f618"
server_key_iv: "9a6887619d2d51235533823324cb11a2"
server_key_version: "encryptionkey@0"
I20260929 20:16:51.488325 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance:
uuid: "cd78e56b0d864cbe83dd1dbd8f239f3b"
format_stamp: "Formatted at 2026-09-29 20:16:51 on dist-test-slave-4cx3"
server_key: "ad8e5b177c2f0084c754c4b0f9c5f618"
server_key_iv: "9a6887619d2d51235533823324cb11a2"
server_key_version: "encryptionkey@0"
I20260929 20:16:51.493276 930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260929 20:16:51.496735 944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:51.498054 930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 20:16:51.498222 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "cd78e56b0d864cbe83dd1dbd8f239f3b"
format_stamp: "Formatted at 2026-09-29 20:16:51 on dist-test-slave-4cx3"
server_key: "ad8e5b177c2f0084c754c4b0f9c5f618"
server_key_iv: "9a6887619d2d51235533823324cb11a2"
server_key_version: "encryptionkey@0"
I20260929 20:16:51.498433 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:51.503793 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:51.504532 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:51.504796 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:51.505661 930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:16:51.507128 930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:16:51.507251 930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:51.507375 930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:16:51.507498 930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260929 20:16:51.540393 930 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:46383
I20260929 20:16:51.540442 1056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:46383 every 8 connection(s)
I20260929 20:16:51.541734 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:16:51.547358 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 930
I20260929 20:16:51.547618 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance
I20260929 20:16:51.548120 28389 external_mini_cluster.cc:1428] Setting key 87a4713d56052aaeed7eee9ad3efdc32
I20260929 20:16:51.550007 798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:51.544339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:46569 (local address 127.27.185.126:41631)
0929 20:16:51.544541 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:51.544546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:51.545885 (+ 1339us) server_negotiation.cc:407] Connection header received
0929 20:16:51.547621 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:51.547629 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:51.547703 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:51.547821 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:51.549164 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:51.549176 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:51.549179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:16:51.549193 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:51.549223 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:16:51.549236 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:51.549358 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:51.549628 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 20:16:51.549694 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:16:51.550942 1059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:51.544885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41631 (local address 127.27.185.67:46569)
0929 20:16:51.545758 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:51.545795 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:51.547282 (+ 1487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:51.548011 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:51.548019 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:51.548107 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:51.548853 (+ 746us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:16:51.548859 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:51.548914 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:16:51.549014 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:51.549345 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:51.549355 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:51.549365 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 20:16:51.549674 (+ 309us) client_negotiation.cc:240] Negotiation successful
0929 20:16:51.549867 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":253,"threads_started":1}
I20260929 20:16:51.552228 1057 heartbeater.cc:347] Connected to a master server at 127.27.185.126:41631
I20260929 20:16:51.552528 1057 heartbeater.cc:464] Registering TS with master...
I20260929 20:16:51.553093 1057 heartbeater.cc:511] Master 127.27.185.126:41631 requested a full tablet report, sending...
I20260929 20:16:51.554409 610 ts_manager.cc:194] Registered new tserver with Master: cd78e56b0d864cbe83dd1dbd8f239f3b (127.27.185.67:46383)
I20260929 20:16:51.563843 28389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:16:51.571017 798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:51.568308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49244 (local address 127.27.185.126:41631)
0929 20:16:51.568474 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:51.568480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:51.568970 (+ 490us) server_negotiation.cc:407] Connection header received
0929 20:16:51.569180 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:51.569185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:51.569237 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:51.569357 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:51.570357 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:51.570363 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:51.570368 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:16:51.570393 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:51.570425 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:16:51.570437 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:51.570568 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:51.570684 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 20:16:51.570745 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:16:51.574855 610 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:16:51.581674 610 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:16:51.598263 1069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:51.593274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:34843 (local address 127.0.0.1:47298)
0929 20:16:51.593919 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:51.593967 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:51.594064 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:51.595097 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:51.595104 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:51.595217 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:51.596080 (+ 863us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:16:51.596086 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:51.596121 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:16:51.596204 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:51.597536 (+ 1332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:51.597547 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:51.597557 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 20:16:51.597755 (+ 198us) client_negotiation.cc:240] Negotiation successful
0929 20:16:51.597851 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":326,"threads_started":1}
I20260929 20:16:51.598672 1070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:51.593519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47298 (local address 127.27.185.65:34843)
0929 20:16:51.594471 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:51.594488 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:51.594519 (+ 31us) server_negotiation.cc:407] Connection header received
0929 20:16:51.594591 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:51.594602 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:51.594845 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:51.595739 (+ 894us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:51.597273 (+ 1534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:51.597284 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:51.597302 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:16:51.597329 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:51.597387 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:16:51.597411 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:51.597553 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:51.597973 (+ 420us) server_negotiation.cc:299] Negotiation successful
0929 20:16:51.598077 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":788,"thread_start_us":675,"threads_started":1}
I20260929 20:16:51.601011 1071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:51.594458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:39667 (local address 127.0.0.1:54956)
0929 20:16:51.595416 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:51.595434 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:51.595532 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:51.598531 (+ 2999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:51.598535 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:51.598558 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:51.599710 (+ 1152us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:16:51.599718 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:51.599735 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:16:51.599786 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:51.600343 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:51.600349 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:51.600354 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 20:16:51.600485 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 20:16:51.600537 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":621,"threads_started":1}
I20260929 20:16:51.601482 1073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:51.595240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:46383 (local address 127.0.0.1:43082)
0929 20:16:51.597707 (+ 2467us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:51.597727 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:51.597817 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:51.599548 (+ 1731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:51.599555 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:51.599585 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:51.600221 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:16:51.600225 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:51.600242 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:16:51.600284 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:51.600699 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:51.600703 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:51.600708 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 20:16:51.600879 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 20:16:51.600931 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2335,"thread_start_us":2190,"threads_started":1}
I20260929 20:16:51.601714 1072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:51.594811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54956 (local address 127.27.185.66:39667)
0929 20:16:51.597998 (+ 3187us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:51.598006 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:51.598033 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:16:51.598098 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:51.598104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:51.598314 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:51.598518 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:51.599987 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:51.599997 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:51.600012 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:16:51.600040 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:51.600100 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:16:51.600121 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:51.600280 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:51.601254 (+ 974us) server_negotiation.cc:299] Negotiation successful
0929 20:16:51.601343 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3008,"thread_start_us":2884,"threads_started":1}
I20260929 20:16:51.601910 1074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:51.595818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43082 (local address 127.27.185.67:46383)
0929 20:16:51.596401 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:51.596407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:51.597928 (+ 1521us) server_negotiation.cc:407] Connection header received
0929 20:16:51.598003 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:51.598011 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:51.599287 (+ 1276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:51.599491 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:51.600473 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:51.600481 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:51.600494 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:16:51.600511 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:51.600568 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:16:51.600586 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:51.600746 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:51.600884 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 20:16:51.600997 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":281,"threads_started":1}
I20260929 20:16:51.603583 731 tablet_service.cc:1467] Processing CreateTablet for tablet da1f42d1a22d4c18b4c4811289552e58 (DEFAULT_TABLE table=test-table [id=4d6ca0da62094914b8d86a051e2458a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:16:51.605100 731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da1f42d1a22d4c18b4c4811289552e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:51.605131 992 tablet_service.cc:1467] Processing CreateTablet for tablet da1f42d1a22d4c18b4c4811289552e58 (DEFAULT_TABLE table=test-table [id=4d6ca0da62094914b8d86a051e2458a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:16:51.605708 862 tablet_service.cc:1467] Processing CreateTablet for tablet da1f42d1a22d4c18b4c4811289552e58 (DEFAULT_TABLE table=test-table [id=4d6ca0da62094914b8d86a051e2458a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:16:51.606249 992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da1f42d1a22d4c18b4c4811289552e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:51.607004 862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da1f42d1a22d4c18b4c4811289552e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:51.616948 1075 tablet_bootstrap.cc:492] T da1f42d1a22d4c18b4c4811289552e58 P cd78e56b0d864cbe83dd1dbd8f239f3b: Bootstrap starting.
I20260929 20:16:51.620138 1076 tablet_bootstrap.cc:492] T da1f42d1a22d4c18b4c4811289552e58 P dee81202602c425cb7c97ec67e5250f8: Bootstrap starting.
I20260929 20:16:51.621937 1075 tablet_bootstrap.cc:654] T da1f42d1a22d4c18b4c4811289552e58 P cd78e56b0d864cbe83dd1dbd8f239f3b: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:51.623222 1075 log.cc:824] T da1f42d1a22d4c18b4c4811289552e58 P cd78e56b0d864cbe83dd1dbd8f239f3b: Log is configured to *not* fsync() on all Append() calls
I20260929 20:16:51.623795 1076 tablet_bootstrap.cc:654] T da1f42d1a22d4c18b4c4811289552e58 P dee81202602c425cb7c97ec67e5250f8: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:51.624809 1077 tablet_bootstrap.cc:492] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1: Bootstrap starting.
I20260929 20:16:51.625543 1076 log.cc:824] T da1f42d1a22d4c18b4c4811289552e58 P dee81202602c425cb7c97ec67e5250f8: Log is configured to *not* fsync() on all Append() calls
I20260929 20:16:51.627529 1075 tablet_bootstrap.cc:492] T da1f42d1a22d4c18b4c4811289552e58 P cd78e56b0d864cbe83dd1dbd8f239f3b: No bootstrap required, opened a new log
I20260929 20:16:51.627801 1075 ts_tablet_manager.cc:1397] T da1f42d1a22d4c18b4c4811289552e58 P cd78e56b0d864cbe83dd1dbd8f239f3b: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260929 20:16:51.628415 1077 tablet_bootstrap.cc:654] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:51.628599 1076 tablet_bootstrap.cc:492] T da1f42d1a22d4c18b4c4811289552e58 P dee81202602c425cb7c97ec67e5250f8: No bootstrap required, opened a new log
I20260929 20:16:51.628872 1076 ts_tablet_manager.cc:1397] T da1f42d1a22d4c18b4c4811289552e58 P dee81202602c425cb7c97ec67e5250f8: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 20:16:51.629688 1077 log.cc:824] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1: Log is configured to *not* fsync() on all Append() calls
I20260929 20:16:51.632716 1077 tablet_bootstrap.cc:492] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1: No bootstrap required, opened a new log
I20260929 20:16:51.632988 1077 ts_tablet_manager.cc:1397] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260929 20:16:51.637718 1075 raft_consensus.cc:348] T da1f42d1a22d4c18b4c4811289552e58 P cd78e56b0d864cbe83dd1dbd8f239f3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee81202602c425cb7c97ec67e5250f8" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 34843 } } peers { permanent_uuid: "31401d46097d4e2ab597c4ef0e32f9d1" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 39667 } } peers { permanent_uuid: "cd78e56b0d864cbe83dd1dbd8f239f3b" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 46383 } }
I20260929 20:16:51.638253 1075 raft_consensus.cc:374] T da1f42d1a22d4c18b4c4811289552e58 P cd78e56b0d864cbe83dd1dbd8f239f3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:51.638351 1075 raft_consensus.cc:729] T da1f42d1a22d4c18b4c4811289552e58 P cd78e56b0d864cbe83dd1dbd8f239f3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd78e56b0d864cbe83dd1dbd8f239f3b, State: Initialized, Role: FOLLOWER
I20260929 20:16:51.639000 1075 consensus_queue.cc:260] T da1f42d1a22d4c18b4c4811289552e58 P cd78e56b0d864cbe83dd1dbd8f239f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee81202602c425cb7c97ec67e5250f8" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 34843 } } peers { permanent_uuid: "31401d46097d4e2ab597c4ef0e32f9d1" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 39667 } } peers { permanent_uuid: "cd78e56b0d864cbe83dd1dbd8f239f3b" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 46383 } }
I20260929 20:16:51.639621 1076 raft_consensus.cc:348] T da1f42d1a22d4c18b4c4811289552e58 P dee81202602c425cb7c97ec67e5250f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee81202602c425cb7c97ec67e5250f8" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 34843 } } peers { permanent_uuid: "31401d46097d4e2ab597c4ef0e32f9d1" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 39667 } } peers { permanent_uuid: "cd78e56b0d864cbe83dd1dbd8f239f3b" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 46383 } }
I20260929 20:16:51.640053 1076 raft_consensus.cc:374] T da1f42d1a22d4c18b4c4811289552e58 P dee81202602c425cb7c97ec67e5250f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:51.640137 1057 heartbeater.cc:503] Master 127.27.185.126:41631 was elected leader, sending a full tablet report...
I20260929 20:16:51.640149 1076 raft_consensus.cc:729] T da1f42d1a22d4c18b4c4811289552e58 P dee81202602c425cb7c97ec67e5250f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee81202602c425cb7c97ec67e5250f8, State: Initialized, Role: FOLLOWER
I20260929 20:16:51.640735 1076 consensus_queue.cc:260] T da1f42d1a22d4c18b4c4811289552e58 P dee81202602c425cb7c97ec67e5250f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee81202602c425cb7c97ec67e5250f8" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 34843 } } peers { permanent_uuid: "31401d46097d4e2ab597c4ef0e32f9d1" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 39667 } } peers { permanent_uuid: "cd78e56b0d864cbe83dd1dbd8f239f3b" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 46383 } }
I20260929 20:16:51.641958 796 heartbeater.cc:503] Master 127.27.185.126:41631 was elected leader, sending a full tablet report...
I20260929 20:16:51.642560 1076 ts_tablet_manager.cc:1428] T da1f42d1a22d4c18b4c4811289552e58 P dee81202602c425cb7c97ec67e5250f8: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260929 20:16:51.643292 1077 raft_consensus.cc:348] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee81202602c425cb7c97ec67e5250f8" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 34843 } } peers { permanent_uuid: "31401d46097d4e2ab597c4ef0e32f9d1" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 39667 } } peers { permanent_uuid: "cd78e56b0d864cbe83dd1dbd8f239f3b" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 46383 } }
I20260929 20:16:51.643723 1075 ts_tablet_manager.cc:1428] T da1f42d1a22d4c18b4c4811289552e58 P cd78e56b0d864cbe83dd1dbd8f239f3b: Time spent starting tablet: real 0.016s user 0.009s sys 0.003s
I20260929 20:16:51.643787 1077 raft_consensus.cc:374] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:51.643872 1077 raft_consensus.cc:729] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31401d46097d4e2ab597c4ef0e32f9d1, State: Initialized, Role: FOLLOWER
I20260929 20:16:51.644477 1077 consensus_queue.cc:260] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee81202602c425cb7c97ec67e5250f8" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 34843 } } peers { permanent_uuid: "31401d46097d4e2ab597c4ef0e32f9d1" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 39667 } } peers { permanent_uuid: "cd78e56b0d864cbe83dd1dbd8f239f3b" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 46383 } }
I20260929 20:16:51.646036 927 heartbeater.cc:503] Master 127.27.185.126:41631 was elected leader, sending a full tablet report...
I20260929 20:16:51.646731 1077 ts_tablet_manager.cc:1428] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1: Time spent starting tablet: real 0.014s user 0.012s sys 0.002s
W20260929 20:16:51.692638 797 tablet.cc:2369] T da1f42d1a22d4c18b4c4811289552e58 P dee81202602c425cb7c97ec67e5250f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:16:51.793398 1058 tablet.cc:2369] T da1f42d1a22d4c18b4c4811289552e58 P cd78e56b0d864cbe83dd1dbd8f239f3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:16:51.802554 1083 raft_consensus.cc:482] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:16:51.802932 1083 raft_consensus.cc:504] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee81202602c425cb7c97ec67e5250f8" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 34843 } } peers { permanent_uuid: "31401d46097d4e2ab597c4ef0e32f9d1" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 39667 } } peers { permanent_uuid: "cd78e56b0d864cbe83dd1dbd8f239f3b" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 46383 } }
I20260929 20:16:51.804657 1083 leader_election.cc:288] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dee81202602c425cb7c97ec67e5250f8 (127.27.185.65:34843), cd78e56b0d864cbe83dd1dbd8f239f3b (127.27.185.67:46383)
I20260929 20:16:51.808090 1070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:51.804964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:44469 (local address 127.27.185.65:34843)
0929 20:16:51.805195 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:51.805204 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:51.805937 (+ 733us) server_negotiation.cc:407] Connection header received
0929 20:16:51.806105 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:51.806113 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:51.806179 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:51.806260 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:51.807362 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:51.807371 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:51.807379 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:16:51.807429 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:51.807469 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:16:51.807488 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:51.807585 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:51.807712 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 20:16:51.807779 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 20:16:51.808122 1074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:51.805088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:40215 (local address 127.27.185.67:46383)
0929 20:16:51.805279 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:51.805287 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:51.806015 (+ 728us) server_negotiation.cc:407] Connection header received
0929 20:16:51.806189 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:51.806193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:51.806267 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:51.806365 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:51.807500 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:51.807504 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:51.807507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:16:51.807521 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:51.807571 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:16:51.807586 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:51.807707 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:51.807827 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 20:16:51.807884 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:16:51.808274 1084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:51.804884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:34843 (local address 127.27.185.66:44469)
0929 20:16:51.805817 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:51.805832 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:51.805944 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:51.806323 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:51.806330 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:51.806354 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:51.807170 (+ 816us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:16:51.807172 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:51.807185 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:16:51.807223 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:51.807584 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:51.807590 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:51.807595 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 20:16:51.807703 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 20:16:51.807780 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"thread_start_us":597,"threads_started":1}
I20260929 20:16:51.808408 1085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:51.805067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:46383 (local address 127.27.185.66:40215)
0929 20:16:51.805913 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:51.805931 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:51.806060 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:51.806413 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:51.806420 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:51.806444 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:51.807251 (+ 807us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:16:51.807254 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:51.807282 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:16:51.807333 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:51.807701 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:51.807705 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:51.807710 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 20:16:51.807852 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 20:16:51.807901 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":493,"threads_started":1}
I20260929 20:16:51.809264 751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1f42d1a22d4c18b4c4811289552e58" candidate_uuid: "31401d46097d4e2ab597c4ef0e32f9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee81202602c425cb7c97ec67e5250f8" is_pre_election: true
I20260929 20:16:51.809340 1012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1f42d1a22d4c18b4c4811289552e58" candidate_uuid: "31401d46097d4e2ab597c4ef0e32f9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd78e56b0d864cbe83dd1dbd8f239f3b" is_pre_election: true
I20260929 20:16:51.809783 751 raft_consensus.cc:2454] T da1f42d1a22d4c18b4c4811289552e58 P dee81202602c425cb7c97ec67e5250f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31401d46097d4e2ab597c4ef0e32f9d1 in term 0.
I20260929 20:16:51.809787 1012 raft_consensus.cc:2454] T da1f42d1a22d4c18b4c4811289552e58 P cd78e56b0d864cbe83dd1dbd8f239f3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31401d46097d4e2ab597c4ef0e32f9d1 in term 0.
I20260929 20:16:51.810461 818 leader_election.cc:302] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 31401d46097d4e2ab597c4ef0e32f9d1, cd78e56b0d864cbe83dd1dbd8f239f3b, dee81202602c425cb7c97ec67e5250f8; no voters:
I20260929 20:16:51.810868 1083 raft_consensus.cc:2781] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:16:51.811009 1083 raft_consensus.cc:482] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:16:51.811112 1083 raft_consensus.cc:3037] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:51.813314 1083 raft_consensus.cc:504] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee81202602c425cb7c97ec67e5250f8" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 34843 } } peers { permanent_uuid: "31401d46097d4e2ab597c4ef0e32f9d1" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 39667 } } peers { permanent_uuid: "cd78e56b0d864cbe83dd1dbd8f239f3b" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 46383 } }
I20260929 20:16:51.813911 1083 leader_election.cc:288] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1 [CANDIDATE]: Term 1 election: Requested vote from peers dee81202602c425cb7c97ec67e5250f8 (127.27.185.65:34843), cd78e56b0d864cbe83dd1dbd8f239f3b (127.27.185.67:46383)
I20260929 20:16:51.814325 751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1f42d1a22d4c18b4c4811289552e58" candidate_uuid: "31401d46097d4e2ab597c4ef0e32f9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee81202602c425cb7c97ec67e5250f8"
I20260929 20:16:51.814392 1012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1f42d1a22d4c18b4c4811289552e58" candidate_uuid: "31401d46097d4e2ab597c4ef0e32f9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd78e56b0d864cbe83dd1dbd8f239f3b"
I20260929 20:16:51.814539 751 raft_consensus.cc:3037] T da1f42d1a22d4c18b4c4811289552e58 P dee81202602c425cb7c97ec67e5250f8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:51.814579 1012 raft_consensus.cc:3037] T da1f42d1a22d4c18b4c4811289552e58 P cd78e56b0d864cbe83dd1dbd8f239f3b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:51.817863 1012 raft_consensus.cc:2454] T da1f42d1a22d4c18b4c4811289552e58 P cd78e56b0d864cbe83dd1dbd8f239f3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31401d46097d4e2ab597c4ef0e32f9d1 in term 1.
I20260929 20:16:51.817867 751 raft_consensus.cc:2454] T da1f42d1a22d4c18b4c4811289552e58 P dee81202602c425cb7c97ec67e5250f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31401d46097d4e2ab597c4ef0e32f9d1 in term 1.
I20260929 20:16:51.818325 816 leader_election.cc:302] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31401d46097d4e2ab597c4ef0e32f9d1, cd78e56b0d864cbe83dd1dbd8f239f3b; no voters:
I20260929 20:16:51.818660 1083 raft_consensus.cc:2781] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:16:51.819161 1083 raft_consensus.cc:686] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1 [term 1 LEADER]: Becoming Leader. State: Replica: 31401d46097d4e2ab597c4ef0e32f9d1, State: Running, Role: LEADER
I20260929 20:16:51.819701 1083 consensus_queue.cc:237] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee81202602c425cb7c97ec67e5250f8" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 34843 } } peers { permanent_uuid: "31401d46097d4e2ab597c4ef0e32f9d1" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 39667 } } peers { permanent_uuid: "cd78e56b0d864cbe83dd1dbd8f239f3b" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 46383 } }
I20260929 20:16:51.824754 610 catalog_manager.cc:5495] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31401d46097d4e2ab597c4ef0e32f9d1 (127.27.185.66). New cstate: current_term: 1 leader_uuid: "31401d46097d4e2ab597c4ef0e32f9d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee81202602c425cb7c97ec67e5250f8" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 34843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31401d46097d4e2ab597c4ef0e32f9d1" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 39667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd78e56b0d864cbe83dd1dbd8f239f3b" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 46383 } health_report { overall_health: UNKNOWN } } }
I20260929 20:16:51.844901 1072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:51.838136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54962 (local address 127.27.185.66:39667)
0929 20:16:51.838309 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:51.838313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:51.838330 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:16:51.838415 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:51.838419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:51.838500 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:51.838581 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:51.844379 (+ 5798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:51.844384 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:51.844387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:16:51.844406 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:51.844444 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:16:51.844456 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:51.844565 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:51.844653 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 20:16:51.844706 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:16:51.854079 751 raft_consensus.cc:1261] T da1f42d1a22d4c18b4c4811289552e58 P dee81202602c425cb7c97ec67e5250f8 [term 1 FOLLOWER]: Refusing update from remote peer 31401d46097d4e2ab597c4ef0e32f9d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:16:51.854082 1012 raft_consensus.cc:1261] T da1f42d1a22d4c18b4c4811289552e58 P cd78e56b0d864cbe83dd1dbd8f239f3b [term 1 FOLLOWER]: Refusing update from remote peer 31401d46097d4e2ab597c4ef0e32f9d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:16:51.855099 1083 consensus_queue.cc:1035] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dee81202602c425cb7c97ec67e5250f8" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 34843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:16:51.855451 1086 consensus_queue.cc:1035] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd78e56b0d864cbe83dd1dbd8f239f3b" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:16:51.882230 609 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:16:51.882535 609 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:16:51.886059 609 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0cc7a445f9ea40db97ca0ab0ab9e6f6a: Sending DeleteTablet for 3 replicas of tablet da1f42d1a22d4c18b4c4811289552e58
I20260929 20:16:51.887166 731 tablet_service.cc:1514] Processing DeleteTablet for tablet da1f42d1a22d4c18b4c4811289552e58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:16:51 UTC) from {username='slave'} at 127.0.0.1:47298
I20260929 20:16:51.887224 992 tablet_service.cc:1514] Processing DeleteTablet for tablet da1f42d1a22d4c18b4c4811289552e58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:16:51 UTC) from {username='slave'} at 127.0.0.1:43082
I20260929 20:16:51.887203 862 tablet_service.cc:1514] Processing DeleteTablet for tablet da1f42d1a22d4c18b4c4811289552e58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:16:51 UTC) from {username='slave'} at 127.0.0.1:54956
I20260929 20:16:51.888309 1098 tablet_replica.cc:330] T da1f42d1a22d4c18b4c4811289552e58 P dee81202602c425cb7c97ec67e5250f8: stopping tablet replica
I20260929 20:16:51.888974 1098 raft_consensus.cc:2229] T da1f42d1a22d4c18b4c4811289552e58 P dee81202602c425cb7c97ec67e5250f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:16:51.889689 1098 raft_consensus.cc:2258] T da1f42d1a22d4c18b4c4811289552e58 P dee81202602c425cb7c97ec67e5250f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:16:51.890659 1099 tablet_replica.cc:330] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1: stopping tablet replica
I20260929 20:16:51.890875 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 669
I20260929 20:16:51.891274 1100 tablet_replica.cc:330] T da1f42d1a22d4c18b4c4811289552e58 P cd78e56b0d864cbe83dd1dbd8f239f3b: stopping tablet replica
I20260929 20:16:51.891786 1100 raft_consensus.cc:2229] T da1f42d1a22d4c18b4c4811289552e58 P cd78e56b0d864cbe83dd1dbd8f239f3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:16:51.892304 1099 raft_consensus.cc:2229] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:16:51.892896 1098 ts_tablet_manager.cc:1905] T da1f42d1a22d4c18b4c4811289552e58 P dee81202602c425cb7c97ec67e5250f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:16:51.893414 1100 raft_consensus.cc:2258] T da1f42d1a22d4c18b4c4811289552e58 P cd78e56b0d864cbe83dd1dbd8f239f3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:16:51.893683 1099 raft_consensus.cc:2258] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:16:51.898490 1100 ts_tablet_manager.cc:1905] T da1f42d1a22d4c18b4c4811289552e58 P cd78e56b0d864cbe83dd1dbd8f239f3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:16:51.900463 1098 ts_tablet_manager.cc:1918] T da1f42d1a22d4c18b4c4811289552e58 P dee81202602c425cb7c97ec67e5250f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:16:51.900627 1098 log.cc:1196] T da1f42d1a22d4c18b4c4811289552e58 P dee81202602c425cb7c97ec67e5250f8: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-0/wal/wals/da1f42d1a22d4c18b4c4811289552e58
I20260929 20:16:51.901156 1098 ts_tablet_manager.cc:1939] T da1f42d1a22d4c18b4c4811289552e58 P dee81202602c425cb7c97ec67e5250f8: Deleting consensus metadata
I20260929 20:16:51.903083 597 catalog_manager.cc:4841] TS dee81202602c425cb7c97ec67e5250f8 (127.27.185.65:34843): tablet da1f42d1a22d4c18b4c4811289552e58 (table test-table [id=4d6ca0da62094914b8d86a051e2458a0]) successfully deleted
I20260929 20:16:51.904343 1099 ts_tablet_manager.cc:1905] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:16:51.907939 1100 ts_tablet_manager.cc:1918] T da1f42d1a22d4c18b4c4811289552e58 P cd78e56b0d864cbe83dd1dbd8f239f3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:16:51.908108 1100 log.cc:1196] T da1f42d1a22d4c18b4c4811289552e58 P cd78e56b0d864cbe83dd1dbd8f239f3b: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-2/wal/wals/da1f42d1a22d4c18b4c4811289552e58
I20260929 20:16:51.908643 1100 ts_tablet_manager.cc:1939] T da1f42d1a22d4c18b4c4811289552e58 P cd78e56b0d864cbe83dd1dbd8f239f3b: Deleting consensus metadata
I20260929 20:16:51.908849 1070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:51.893615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47300 (local address 127.27.185.65:34843)
0929 20:16:51.893798 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:51.893804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:51.902452 (+ 8648us) server_negotiation.cc:407] Connection header received
0929 20:16:51.902756 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:51.902764 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:51.902877 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:51.902963 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:51.907878 (+ 4915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:51.907885 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:51.907890 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:16:51.907912 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:51.907953 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:16:51.907970 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:51.908082 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:51.908442 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 20:16:51.908520 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:16:51.910061 791 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:16:51.910519 595 catalog_manager.cc:4841] TS cd78e56b0d864cbe83dd1dbd8f239f3b (127.27.185.67:46383): tablet da1f42d1a22d4c18b4c4811289552e58 (table test-table [id=4d6ca0da62094914b8d86a051e2458a0]) successfully deleted
I20260929 20:16:51.911803 1099 ts_tablet_manager.cc:1918] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:16:51.911988 1099 log.cc:1196] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790712966917858-28389-0/minicluster-data/ts-1/wal/wals/da1f42d1a22d4c18b4c4811289552e58
I20260929 20:16:51.912456 1099 ts_tablet_manager.cc:1939] T da1f42d1a22d4c18b4c4811289552e58 P 31401d46097d4e2ab597c4ef0e32f9d1: Deleting consensus metadata
I20260929 20:16:51.913887 597 catalog_manager.cc:4841] TS 31401d46097d4e2ab597c4ef0e32f9d1 (127.27.185.66:39667): tablet da1f42d1a22d4c18b4c4811289552e58 (table test-table [id=4d6ca0da62094914b8d86a051e2458a0]) successfully deleted
I20260929 20:16:52.001950 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 669
I20260929 20:16:52.020097 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 800
I20260929 20:16:52.023116 1072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:52.021059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54976 (local address 127.27.185.66:39667)
0929 20:16:52.021236 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:52.021240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:52.021257 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:16:52.021333 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:52.021337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:52.021387 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:52.021478 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:52.022391 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:52.022396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:52.022399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:16:52.022414 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:52.022467 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:16:52.022479 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:52.022590 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:52.022727 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 20:16:52.022778 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:16:52.024006 922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:16:52.160529 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 800
I20260929 20:16:52.176194 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 930
I20260929 20:16:52.178949 1074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:52.176936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43098 (local address 127.27.185.67:46383)
0929 20:16:52.177160 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:52.177164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:52.177208 (+ 44us) server_negotiation.cc:407] Connection header received
0929 20:16:52.177378 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:52.177382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:52.177481 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:52.177566 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:52.178341 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:52.178346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:52.178349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:16:52.178363 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:52.178394 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:16:52.178406 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:52.178518 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:52.178584 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 20:16:52.178629 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260929 20:16:52.179625 1052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:16:52.266669 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 930
I20260929 20:16:52.282637 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 579
I20260929 20:16:52.283875 640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:16:52.390419 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 579
2026-09-29T20:16:52Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2714 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 20:16:52.434034 28389 mini_oidc.cc:152] Starting JWKS server
I20260929 20:16:52.436651 28389 webserver.cc:460] Webserver started at https://127.0.0.1:33931/ using document root <none> and password file <none>
I20260929 20:16:52.436964 28389 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 20:16:52.437625 28389 webserver.cc:460] Webserver started at http://127.0.0.1:46431/ using document root <none> and password file <none>
2026-09-29T20:16:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:16:52Z Disabled control of system clock
I20260929 20:16:52.464797 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:44879
--webserver_interface=127.27.185.126
--webserver_port=0
--builtin_ntp_servers=127.27.185.84:39839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:44879
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33931/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790712966917858-28389-0/testchainca.cert with env {}
W20260929 20:16:52.795013 1111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:52.795320 1111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:52.795469 1111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:52.802731 1111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:16:52.802842 1111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:52.802886 1111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:52.802942 1111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:16:52.802991 1111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 20:16:52.803049 1111 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 20:16:52.803089 1111 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33931/jwks.json
I20260929 20:16:52.813313 1111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:39839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:44879
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:44879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33931/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790712966917858-28389-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:52.814625 1111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:52.815935 1111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:52.828157 1116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:52.828132 1117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:52.828070 1119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:52.829263 1111 server_base.cc:1034] running on GCE node
I20260929 20:16:52.830089 1111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:52.831605 1111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:52.832839 1111 hybrid_clock.cc:648] HybridClock initialized: now 1790713012832787 us; error 41 us; skew 500 ppm
I20260929 20:16:52.833207 1111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:16:52.835510 1111 webserver.cc:460] Webserver started at http://127.27.185.126:39861/ using document root <none> and password file <none>
I20260929 20:16:52.836201 1111 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:52.836333 1111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:52.836629 1111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:52.839563 1111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "1fec16df2e844ef9b5aa50fd7c605180"
format_stamp: "Formatted at 2026-09-29 20:16:52 on dist-test-slave-4cx3"
server_key: "a795e4ac837e789c653053042ca79382"
server_key_iv: "f1fc4022cf14aa39848a68f81b46b0a1"
server_key_version: "encryptionkey@0"
I20260929 20:16:52.840359 1111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "1fec16df2e844ef9b5aa50fd7c605180"
format_stamp: "Formatted at 2026-09-29 20:16:52 on dist-test-slave-4cx3"
server_key: "a795e4ac837e789c653053042ca79382"
server_key_iv: "f1fc4022cf14aa39848a68f81b46b0a1"
server_key_version: "encryptionkey@0"
I20260929 20:16:52.845589 1111 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:16:52.849140 1125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:52.850586 1111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:16:52.850795 1111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "1fec16df2e844ef9b5aa50fd7c605180"
format_stamp: "Formatted at 2026-09-29 20:16:52 on dist-test-slave-4cx3"
server_key: "a795e4ac837e789c653053042ca79382"
server_key_iv: "f1fc4022cf14aa39848a68f81b46b0a1"
server_key_version: "encryptionkey@0"
I20260929 20:16:52.851038 1111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:52.882598 1111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:52.883559 1111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:52.883890 1111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:52.902467 1111 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:44879
I20260929 20:16:52.902526 1176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:44879 every 8 connection(s)
I20260929 20:16:52.903872 1111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:16:52.904261 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 1111
I20260929 20:16:52.904449 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:16:52.905290 28389 external_mini_cluster.cc:1428] Setting key 8dbfce86a95452b64f1a792e068db9a8
I20260929 20:16:52.908349 1177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:52.918865 1180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:52.907284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50434 (local address 127.27.185.126:44879)
0929 20:16:52.908260 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:52.908278 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:52.908319 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:16:52.910078 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:52.910100 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:52.910654 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:52.911082 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:52.911764 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:52.913129 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:52.914476 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:52.915126 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:52.915329 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:52.916575 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:52.916582 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:52.916599 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:16:52.917164 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:52.917238 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:16:52.917258 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:52.917392 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:52.917615 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 20:16:52.917922 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":326,"threads_started":1}
I20260929 20:16:52.918919 1177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fec16df2e844ef9b5aa50fd7c605180: Bootstrap starting.
I20260929 20:16:52.922276 1177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fec16df2e844ef9b5aa50fd7c605180: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:52.922801 28389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:16:52.923471 1177 log.cc:824] T 00000000000000000000000000000000 P 1fec16df2e844ef9b5aa50fd7c605180: Log is configured to *not* fsync() on all Append() calls
I20260929 20:16:52.926636 1177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fec16df2e844ef9b5aa50fd7c605180: No bootstrap required, opened a new log
I20260929 20:16:52.934259 1177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fec16df2e844ef9b5aa50fd7c605180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fec16df2e844ef9b5aa50fd7c605180" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 44879 } }
I20260929 20:16:52.934638 1177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fec16df2e844ef9b5aa50fd7c605180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:52.934736 1177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fec16df2e844ef9b5aa50fd7c605180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fec16df2e844ef9b5aa50fd7c605180, State: Initialized, Role: FOLLOWER
I20260929 20:16:52.935297 1177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fec16df2e844ef9b5aa50fd7c605180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fec16df2e844ef9b5aa50fd7c605180" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 44879 } }
I20260929 20:16:52.935595 1177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fec16df2e844ef9b5aa50fd7c605180 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:16:52.935722 1177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fec16df2e844ef9b5aa50fd7c605180 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:16:52.935909 1177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fec16df2e844ef9b5aa50fd7c605180 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:52.938231 1177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fec16df2e844ef9b5aa50fd7c605180 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fec16df2e844ef9b5aa50fd7c605180" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 44879 } }
I20260929 20:16:52.938740 1177 leader_election.cc:302] T 00000000000000000000000000000000 P 1fec16df2e844ef9b5aa50fd7c605180 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fec16df2e844ef9b5aa50fd7c605180; no voters:
I20260929 20:16:52.939307 1177 leader_election.cc:288] T 00000000000000000000000000000000 P 1fec16df2e844ef9b5aa50fd7c605180 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:16:52.939496 1182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fec16df2e844ef9b5aa50fd7c605180 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:16:52.940291 1182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fec16df2e844ef9b5aa50fd7c605180 [term 1 LEADER]: Becoming Leader. State: Replica: 1fec16df2e844ef9b5aa50fd7c605180, State: Running, Role: LEADER
I20260929 20:16:52.940817 1182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fec16df2e844ef9b5aa50fd7c605180 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fec16df2e844ef9b5aa50fd7c605180" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 44879 } }
I20260929 20:16:52.941532 1177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fec16df2e844ef9b5aa50fd7c605180 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:16:52.943902 1184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fec16df2e844ef9b5aa50fd7c605180 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fec16df2e844ef9b5aa50fd7c605180. Latest consensus state: current_term: 1 leader_uuid: "1fec16df2e844ef9b5aa50fd7c605180" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fec16df2e844ef9b5aa50fd7c605180" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 44879 } } }
I20260929 20:16:52.944415 1184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fec16df2e844ef9b5aa50fd7c605180 [sys.catalog]: This master's current role is: LEADER
I20260929 20:16:52.948819 1191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:16:52.949113 1183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fec16df2e844ef9b5aa50fd7c605180 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fec16df2e844ef9b5aa50fd7c605180" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fec16df2e844ef9b5aa50fd7c605180" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 44879 } } }
I20260929 20:16:52.949463 1183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fec16df2e844ef9b5aa50fd7c605180 [sys.catalog]: This master's current role is: LEADER
I20260929 20:16:52.953656 1191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:16:52.962455 1191 catalog_manager.cc:1347] Generated new cluster ID: 33b977ceb06e4ce4aae74103e844acd0
I20260929 20:16:52.962594 1191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:16:52.980245 1191 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:16:52.981601 1191 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:16:52.988129 1191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fec16df2e844ef9b5aa50fd7c605180: Generated new TSK 0
I20260929 20:16:52.989041 1191 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 20:16:53.029654 1207 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 20:16:53.029839 1207 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 20:16:53.030769 1180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:53.002325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50438 (local address 127.27.185.126:44879)
0929 20:16:53.002572 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:53.002582 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:53.003185 (+ 603us) server_negotiation.cc:407] Connection header received
0929 20:16:53.003571 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:53.003576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:53.003660 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:53.003807 (+ 147us) server_negotiation.cc:226] Negotiated authn=JWT
0929 20:16:53.004396 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:53.005134 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:53.006560 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:53.007007 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:53.007153 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:53.007727 (+ 574us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 20:16:53.030027 (+ 22300us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33931/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 20:16:53.030419 (+ 392us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50438: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33931/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 20:16:53.030917 1180 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50438: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33931/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 20:16:53.032397 1206 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:16:53.002222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:44879 (local address 127.0.0.1:50438)
0929 20:16:53.003073 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:53.003093 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:53.003294 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:53.003841 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:53.003846 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:53.003885 (+ 39us) client_negotiation.cc:189] Negotiated authn=JWT
0929 20:16:53.004232 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:53.004241 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:53.005279 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:53.005286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:53.006416 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:53.006428 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:53.007185 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:53.007191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:53.007339 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:53.007446 (+ 107us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 20:16:53.030364 (+ 22918us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33931/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 20:16:53.030741 (+ 377us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.185.126:44879: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33931/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":543,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.27.185.126:44879: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33931/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 20:16:53.034601 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 1111
I20260929 20:16:53.035969 1172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:16:53.128837 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 1111
2026-09-29T20:16:53Z chronyd exiting
I20260929 20:16:53.179466 28389 test_util.cc:175] -----------------------------------------------
I20260929 20:16:53.179597 28389 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790712966917858-28389-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (751 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 20:16:53.183151 28389 mini_oidc.cc:152] Starting JWKS server
I20260929 20:16:53.185006 28389 webserver.cc:460] Webserver started at https://127.0.0.1:33869/ using document root <none> and password file <none>
I20260929 20:16:53.185204 28389 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 20:16:53.185740 28389 webserver.cc:460] Webserver started at http://127.0.0.1:38945/ using document root <none> and password file <none>
2026-09-29T20:16:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:16:53Z Disabled control of system clock
I20260929 20:16:53.211493 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:41589
--webserver_interface=127.27.185.126
--webserver_port=0
--builtin_ntp_servers=127.27.185.84:39933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:41589
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33869/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790712966917858-28389-0/testchainca.cert with env {}
W20260929 20:16:53.538664 1214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:53.538969 1214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:53.539057 1214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:53.547119 1214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:16:53.547232 1214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:53.547288 1214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:53.547350 1214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:16:53.547462 1214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 20:16:53.547574 1214 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 20:16:53.547633 1214 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33869/jwks.json
I20260929 20:16:53.558394 1214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:39933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:41589
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:41589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33869/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790712966917858-28389-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:53.560055 1214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:53.561611 1214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:53.575254 1219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:53.575299 1222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:53.575377 1220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:53.576084 1214 server_base.cc:1034] running on GCE node
I20260929 20:16:53.576959 1214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:53.578430 1214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:53.579674 1214 hybrid_clock.cc:648] HybridClock initialized: now 1790713013579634 us; error 35 us; skew 500 ppm
I20260929 20:16:53.580097 1214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:16:53.582460 1214 webserver.cc:460] Webserver started at http://127.27.185.126:40827/ using document root <none> and password file <none>
I20260929 20:16:53.583214 1214 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:53.583355 1214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:53.583734 1214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:53.586936 1214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "6f1b6df79cec4f59bd92a737eb514fe0"
format_stamp: "Formatted at 2026-09-29 20:16:53 on dist-test-slave-4cx3"
server_key: "390c6fc75777a6ae1e8b0085dcc6f652"
server_key_iv: "186a17a43ea5c826846b74e07c7705d4"
server_key_version: "encryptionkey@0"
I20260929 20:16:53.587913 1214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "6f1b6df79cec4f59bd92a737eb514fe0"
format_stamp: "Formatted at 2026-09-29 20:16:53 on dist-test-slave-4cx3"
server_key: "390c6fc75777a6ae1e8b0085dcc6f652"
server_key_iv: "186a17a43ea5c826846b74e07c7705d4"
server_key_version: "encryptionkey@0"
I20260929 20:16:53.593181 1214 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:16:53.596719 1228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:53.598138 1214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:16:53.598347 1214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "6f1b6df79cec4f59bd92a737eb514fe0"
format_stamp: "Formatted at 2026-09-29 20:16:53 on dist-test-slave-4cx3"
server_key: "390c6fc75777a6ae1e8b0085dcc6f652"
server_key_iv: "186a17a43ea5c826846b74e07c7705d4"
server_key_version: "encryptionkey@0"
I20260929 20:16:53.598589 1214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:53.625597 1214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:53.626555 1214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:53.626888 1214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:53.646054 1214 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:41589
I20260929 20:16:53.646075 1279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:41589 every 8 connection(s)
I20260929 20:16:53.647507 1214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:16:53.650156 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 1214
I20260929 20:16:53.650379 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:16:53.651158 28389 external_mini_cluster.cc:1428] Setting key 132645ed7d5d8c8434a12aaff6ecdc78
I20260929 20:16:53.651882 1280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:53.664841 1280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f1b6df79cec4f59bd92a737eb514fe0: Bootstrap starting.
I20260929 20:16:53.666414 1283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:53.653282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47262 (local address 127.27.185.126:41589)
0929 20:16:53.654328 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:53.654340 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:53.654390 (+ 50us) server_negotiation.cc:407] Connection header received
0929 20:16:53.656713 (+ 2323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:53.656739 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:53.657296 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:53.657777 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:53.658447 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:53.660039 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:53.661490 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:53.662332 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:53.662562 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:53.663980 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:53.663989 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:53.664004 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:16:53.664587 (+ 583us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:53.664672 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:16:53.664694 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:53.664840 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:53.665068 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 20:16:53.665398 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":353,"threads_started":1}
I20260929 20:16:53.668499 1280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f1b6df79cec4f59bd92a737eb514fe0: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:53.669775 1280 log.cc:824] T 00000000000000000000000000000000 P 6f1b6df79cec4f59bd92a737eb514fe0: Log is configured to *not* fsync() on all Append() calls
I20260929 20:16:53.670370 28389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:16:53.672513 1280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f1b6df79cec4f59bd92a737eb514fe0: No bootstrap required, opened a new log
I20260929 20:16:53.680099 1280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f1b6df79cec4f59bd92a737eb514fe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1b6df79cec4f59bd92a737eb514fe0" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41589 } }
I20260929 20:16:53.680474 1280 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f1b6df79cec4f59bd92a737eb514fe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:53.680549 1280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f1b6df79cec4f59bd92a737eb514fe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f1b6df79cec4f59bd92a737eb514fe0, State: Initialized, Role: FOLLOWER
I20260929 20:16:53.681118 1280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f1b6df79cec4f59bd92a737eb514fe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1b6df79cec4f59bd92a737eb514fe0" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41589 } }
I20260929 20:16:53.681319 1280 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f1b6df79cec4f59bd92a737eb514fe0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:16:53.681419 1280 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f1b6df79cec4f59bd92a737eb514fe0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:16:53.681537 1280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f1b6df79cec4f59bd92a737eb514fe0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:53.683957 1280 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f1b6df79cec4f59bd92a737eb514fe0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1b6df79cec4f59bd92a737eb514fe0" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41589 } }
I20260929 20:16:53.684475 1280 leader_election.cc:302] T 00000000000000000000000000000000 P 6f1b6df79cec4f59bd92a737eb514fe0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f1b6df79cec4f59bd92a737eb514fe0; no voters:
I20260929 20:16:53.685019 1280 leader_election.cc:288] T 00000000000000000000000000000000 P 6f1b6df79cec4f59bd92a737eb514fe0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:16:53.685189 1285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f1b6df79cec4f59bd92a737eb514fe0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:16:53.685964 1285 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f1b6df79cec4f59bd92a737eb514fe0 [term 1 LEADER]: Becoming Leader. State: Replica: 6f1b6df79cec4f59bd92a737eb514fe0, State: Running, Role: LEADER
I20260929 20:16:53.686473 1285 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f1b6df79cec4f59bd92a737eb514fe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1b6df79cec4f59bd92a737eb514fe0" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41589 } }
I20260929 20:16:53.687139 1280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f1b6df79cec4f59bd92a737eb514fe0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:16:53.689390 1287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f1b6df79cec4f59bd92a737eb514fe0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f1b6df79cec4f59bd92a737eb514fe0. Latest consensus state: current_term: 1 leader_uuid: "6f1b6df79cec4f59bd92a737eb514fe0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1b6df79cec4f59bd92a737eb514fe0" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41589 } } }
I20260929 20:16:53.689713 1287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f1b6df79cec4f59bd92a737eb514fe0 [sys.catalog]: This master's current role is: LEADER
I20260929 20:16:53.690245 1286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f1b6df79cec4f59bd92a737eb514fe0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f1b6df79cec4f59bd92a737eb514fe0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1b6df79cec4f59bd92a737eb514fe0" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41589 } } }
I20260929 20:16:53.690608 1291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:16:53.690680 1286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f1b6df79cec4f59bd92a737eb514fe0 [sys.catalog]: This master's current role is: LEADER
I20260929 20:16:53.697012 1291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:16:53.708161 1291 catalog_manager.cc:1347] Generated new cluster ID: a5be55163a5b4cd89652ed31d205ce11
I20260929 20:16:53.708299 1291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:16:53.742940 1291 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:16:53.744334 1291 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:16:53.754191 1291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f1b6df79cec4f59bd92a737eb514fe0: Generated new TSK 0
I20260929 20:16:53.755055 1291 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 20:16:53.794085 1310 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 20:16:53.794219 1310 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 20:16:53.795276 1283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:53.775179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47270 (local address 127.27.185.126:41589)
0929 20:16:53.775436 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:53.775444 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:53.775788 (+ 344us) server_negotiation.cc:407] Connection header received
0929 20:16:53.776041 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:53.776045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:53.776118 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:53.776245 (+ 127us) server_negotiation.cc:226] Negotiated authn=JWT
0929 20:16:53.776819 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:53.777556 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:53.779592 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:53.780061 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:53.780156 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:53.780742 (+ 586us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 20:16:53.794725 (+ 13983us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 20:16:53.795039 (+ 314us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47270: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":80}
W20260929 20:16:53.795470 1283 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47270: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 20:16:53.795732 1309 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:16:53.775013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41589 (local address 127.0.0.1:47270)
0929 20:16:53.775677 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:53.775697 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:53.775851 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:53.776291 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:53.776296 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:53.776310 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0929 20:16:53.776643 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:53.776655 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:53.777709 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:53.777715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:53.779446 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:53.779463 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:53.780230 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:53.780237 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:53.780445 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:53.780493 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 20:16:53.794984 (+ 14491us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 20:16:53.795283 (+ 299us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.185.126:41589: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":337,"threads_started":1}
I20260929 20:16:53.797385 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 1214
I20260929 20:16:53.798588 1275 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:16:53.896207 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 1214
2026-09-29T20:16:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (768 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 20:16:53.951644 28389 mini_oidc.cc:152] Starting JWKS server
I20260929 20:16:53.953593 28389 webserver.cc:460] Webserver started at https://127.0.0.1:34453/ using document root <none> and password file <none>
I20260929 20:16:53.953832 28389 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 20:16:53.954474 28389 webserver.cc:460] Webserver started at http://127.0.0.1:39045/ using document root <none> and password file <none>
2026-09-29T20:16:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:16:53Z Disabled control of system clock
I20260929 20:16:53.981817 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:34441
--webserver_interface=127.27.185.126
--webserver_port=0
--builtin_ntp_servers=127.27.185.84:34017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:34441
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34453/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 20:16:54.306357 1317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:54.306713 1317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:54.306880 1317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:54.314088 1317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:16:54.314267 1317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:54.314369 1317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:54.314469 1317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:16:54.314551 1317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 20:16:54.314646 1317 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 20:16:54.314705 1317 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34453/jwks.json
I20260929 20:16:54.324028 1317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:34017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:34441
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:34441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34453/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:54.325423 1317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:54.327229 1317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:54.339178 1322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:54.339133 1323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:54.339211 1325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:54.340066 1317 server_base.cc:1034] running on GCE node
I20260929 20:16:54.340809 1317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:54.342159 1317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:54.343529 1317 hybrid_clock.cc:648] HybridClock initialized: now 1790713014343427 us; error 61 us; skew 500 ppm
I20260929 20:16:54.343890 1317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:16:54.346182 1317 webserver.cc:460] Webserver started at http://127.27.185.126:45275/ using document root <none> and password file <none>
I20260929 20:16:54.346885 1317 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:54.347036 1317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:54.347460 1317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:54.350427 1317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "b4c7cd582f6c448b93baf0fab44c7d58"
format_stamp: "Formatted at 2026-09-29 20:16:54 on dist-test-slave-4cx3"
server_key: "e5179cda609f2f9a4c16daa513953d53"
server_key_iv: "a66d04369b25e0d1b51487c37e7e1b74"
server_key_version: "encryptionkey@0"
I20260929 20:16:54.351236 1317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "b4c7cd582f6c448b93baf0fab44c7d58"
format_stamp: "Formatted at 2026-09-29 20:16:54 on dist-test-slave-4cx3"
server_key: "e5179cda609f2f9a4c16daa513953d53"
server_key_iv: "a66d04369b25e0d1b51487c37e7e1b74"
server_key_version: "encryptionkey@0"
I20260929 20:16:54.356765 1317 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 20:16:54.360639 1331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:54.362226 1317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:16:54.362411 1317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "b4c7cd582f6c448b93baf0fab44c7d58"
format_stamp: "Formatted at 2026-09-29 20:16:54 on dist-test-slave-4cx3"
server_key: "e5179cda609f2f9a4c16daa513953d53"
server_key_iv: "a66d04369b25e0d1b51487c37e7e1b74"
server_key_version: "encryptionkey@0"
I20260929 20:16:54.362609 1317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:54.378837 1317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:54.379691 1317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:54.380046 1317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:54.398995 1317 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:34441
I20260929 20:16:54.399009 1382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:34441 every 8 connection(s)
I20260929 20:16:54.400454 1317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:16:54.400861 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 1317
I20260929 20:16:54.401041 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:16:54.401887 28389 external_mini_cluster.cc:1428] Setting key cf3db6f04ab505b0663cf08f39bf1779
I20260929 20:16:54.405118 1383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:54.415040 1386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:54.403904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56700 (local address 127.27.185.126:34441)
0929 20:16:54.404868 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:54.404879 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:54.404919 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:16:54.406518 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:54.406540 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:54.406972 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:54.407410 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:54.407992 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:54.409291 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:54.410623 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:54.411297 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:54.411568 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:54.412756 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:54.412763 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:54.412779 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:16:54.413364 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:54.413442 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:16:54.413462 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:54.413617 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:54.413840 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 20:16:54.414143 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":352,"threads_started":1}
I20260929 20:16:54.416164 1383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4c7cd582f6c448b93baf0fab44c7d58: Bootstrap starting.
I20260929 20:16:54.418841 28389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:16:54.419555 1383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4c7cd582f6c448b93baf0fab44c7d58: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:54.420513 1383 log.cc:824] T 00000000000000000000000000000000 P b4c7cd582f6c448b93baf0fab44c7d58: Log is configured to *not* fsync() on all Append() calls
I20260929 20:16:54.423249 1383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4c7cd582f6c448b93baf0fab44c7d58: No bootstrap required, opened a new log
I20260929 20:16:54.430312 1383 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4c7cd582f6c448b93baf0fab44c7d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c7cd582f6c448b93baf0fab44c7d58" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34441 } }
I20260929 20:16:54.430665 1383 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4c7cd582f6c448b93baf0fab44c7d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:54.430760 1383 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4c7cd582f6c448b93baf0fab44c7d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c7cd582f6c448b93baf0fab44c7d58, State: Initialized, Role: FOLLOWER
I20260929 20:16:54.431345 1383 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4c7cd582f6c448b93baf0fab44c7d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c7cd582f6c448b93baf0fab44c7d58" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34441 } }
I20260929 20:16:54.431653 1383 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4c7cd582f6c448b93baf0fab44c7d58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:16:54.431788 1383 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4c7cd582f6c448b93baf0fab44c7d58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:16:54.431979 1383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4c7cd582f6c448b93baf0fab44c7d58 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:54.434051 1383 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4c7cd582f6c448b93baf0fab44c7d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c7cd582f6c448b93baf0fab44c7d58" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34441 } }
I20260929 20:16:54.434544 1383 leader_election.cc:302] T 00000000000000000000000000000000 P b4c7cd582f6c448b93baf0fab44c7d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4c7cd582f6c448b93baf0fab44c7d58; no voters:
I20260929 20:16:54.435114 1383 leader_election.cc:288] T 00000000000000000000000000000000 P b4c7cd582f6c448b93baf0fab44c7d58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:16:54.435292 1388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4c7cd582f6c448b93baf0fab44c7d58 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:16:54.436084 1388 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4c7cd582f6c448b93baf0fab44c7d58 [term 1 LEADER]: Becoming Leader. State: Replica: b4c7cd582f6c448b93baf0fab44c7d58, State: Running, Role: LEADER
I20260929 20:16:54.436599 1388 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4c7cd582f6c448b93baf0fab44c7d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c7cd582f6c448b93baf0fab44c7d58" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34441 } }
I20260929 20:16:54.437368 1383 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4c7cd582f6c448b93baf0fab44c7d58 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:16:54.439555 1390 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4c7cd582f6c448b93baf0fab44c7d58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4c7cd582f6c448b93baf0fab44c7d58. Latest consensus state: current_term: 1 leader_uuid: "b4c7cd582f6c448b93baf0fab44c7d58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c7cd582f6c448b93baf0fab44c7d58" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34441 } } }
I20260929 20:16:54.439615 1389 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4c7cd582f6c448b93baf0fab44c7d58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4c7cd582f6c448b93baf0fab44c7d58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c7cd582f6c448b93baf0fab44c7d58" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34441 } } }
I20260929 20:16:54.440078 1390 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4c7cd582f6c448b93baf0fab44c7d58 [sys.catalog]: This master's current role is: LEADER
I20260929 20:16:54.440074 1389 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4c7cd582f6c448b93baf0fab44c7d58 [sys.catalog]: This master's current role is: LEADER
I20260929 20:16:54.442035 1396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:16:54.446759 1396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:16:54.454948 1396 catalog_manager.cc:1347] Generated new cluster ID: 4baeca8cc643482880767af4b57650bd
I20260929 20:16:54.455065 1396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:16:54.474642 1396 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:16:54.476028 1396 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:16:54.488056 1396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4c7cd582f6c448b93baf0fab44c7d58: Generated new TSK 0
I20260929 20:16:54.488934 1396 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:16:54.511258 1386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:54.502665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56712 (local address 127.27.185.126:34441)
0929 20:16:54.502907 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:54.502914 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:54.503562 (+ 648us) server_negotiation.cc:407] Connection header received
0929 20:16:54.503842 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:54.503846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:54.503918 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:54.504100 (+ 182us) server_negotiation.cc:226] Negotiated authn=JWT
0929 20:16:54.504606 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:54.505327 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:54.506704 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:54.507160 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:54.507321 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:54.508211 (+ 890us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 20:16:54.510577 (+ 2366us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34453/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 20:16:54.510920 (+ 343us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56712: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34453/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":75}
W20260929 20:16:54.511677 1386 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56712: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34453/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 20:16:54.512897 1412 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:16:54.502623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:34441 (local address 127.0.0.1:56712)
0929 20:16:54.503433 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:54.503454 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:54.503641 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:54.504111 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:54.504115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:54.504127 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0929 20:16:54.504445 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:54.504454 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:54.505485 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:54.505491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:54.506560 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:54.506571 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:54.507358 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:54.507365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:54.507547 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:54.507580 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 20:16:54.512206 (+ 4626us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34453/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 20:16:54.512487 (+ 281us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.185.126:34441: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34453/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":492,"threads_started":1}
I20260929 20:16:54.514770 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 1317
I20260929 20:16:54.515983 1378 generic_service.cc:196] Checking for leaks (request via RPC)
E20260929 20:16:54.534045 1413 webserver.cc:533] Webserver: sslize failed:
E20260929 20:16:54.534183 1413 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 20:16:54.625346 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 1317
2026-09-29T20:16:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (729 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T20:16:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:16:54Z Disabled control of system clock
I20260929 20:16:54.708117 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:46725
--webserver_interface=127.27.185.126
--webserver_port=0
--builtin_ntp_servers=127.27.185.84:40657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:46725
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790712966917858-28389-0/testchainca.cert with env {}
W20260929 20:16:55.031838 1418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:55.032173 1418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:55.032266 1418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:55.039740 1418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:16:55.039860 1418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:55.039897 1418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:55.039960 1418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:16:55.040005 1418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 20:16:55.040066 1418 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 20:16:55.040107 1418 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 20:16:55.050181 1418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:40657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:46725
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:46725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790712966917858-28389-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:55.051558 1418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:55.052853 1418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:55.066061 1424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:55.066146 1426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:55.066324 1423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:55.066839 1418 server_base.cc:1034] running on GCE node
I20260929 20:16:55.067869 1418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:55.069525 1418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:55.070780 1418 hybrid_clock.cc:648] HybridClock initialized: now 1790713015070751 us; error 53 us; skew 500 ppm
I20260929 20:16:55.071164 1418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:16:55.073695 1418 webserver.cc:460] Webserver started at http://127.27.185.126:37197/ using document root <none> and password file <none>
I20260929 20:16:55.074435 1418 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:55.074577 1418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:55.074908 1418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:55.077980 1418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "b1c48f1a06bf41d280b0bb9ee44a6a4c"
format_stamp: "Formatted at 2026-09-29 20:16:55 on dist-test-slave-4cx3"
server_key: "7c171aa59ef227e5c2316bd408670a3c"
server_key_iv: "b75054534681ed51414763ff7fac735d"
server_key_version: "encryptionkey@0"
I20260929 20:16:55.078816 1418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "b1c48f1a06bf41d280b0bb9ee44a6a4c"
format_stamp: "Formatted at 2026-09-29 20:16:55 on dist-test-slave-4cx3"
server_key: "7c171aa59ef227e5c2316bd408670a3c"
server_key_iv: "b75054534681ed51414763ff7fac735d"
server_key_version: "encryptionkey@0"
I20260929 20:16:55.084141 1418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260929 20:16:55.087564 1432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:55.088850 1418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:16:55.089036 1418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "b1c48f1a06bf41d280b0bb9ee44a6a4c"
format_stamp: "Formatted at 2026-09-29 20:16:55 on dist-test-slave-4cx3"
server_key: "7c171aa59ef227e5c2316bd408670a3c"
server_key_iv: "b75054534681ed51414763ff7fac735d"
server_key_version: "encryptionkey@0"
I20260929 20:16:55.089257 1418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:55.127108 1418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:55.128080 1418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:55.128386 1418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:55.146929 1418 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:46725
I20260929 20:16:55.147024 1483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:46725 every 8 connection(s)
I20260929 20:16:55.148322 1418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:16:55.152266 1484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:16:55.157544 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 1418
I20260929 20:16:55.157759 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:16:55.158532 28389 external_mini_cluster.cc:1428] Setting key 563d308fb4d80dcfe81b41fe224d2016
I20260929 20:16:55.164453 1484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1c48f1a06bf41d280b0bb9ee44a6a4c: Bootstrap starting.
I20260929 20:16:55.168520 1484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1c48f1a06bf41d280b0bb9ee44a6a4c: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:55.169755 1484 log.cc:824] T 00000000000000000000000000000000 P b1c48f1a06bf41d280b0bb9ee44a6a4c: Log is configured to *not* fsync() on all Append() calls
I20260929 20:16:55.173141 1484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1c48f1a06bf41d280b0bb9ee44a6a4c: No bootstrap required, opened a new log
I20260929 20:16:55.176354 1487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:55.160908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39088 (local address 127.27.185.126:46725)
0929 20:16:55.163174 (+ 2266us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:55.163190 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:55.163319 (+ 129us) server_negotiation.cc:407] Connection header received
0929 20:16:55.165743 (+ 2424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:55.165770 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:55.166422 (+ 652us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:55.166883 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:55.167440 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:55.169421 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:55.171069 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:55.172137 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:55.172341 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:55.173416 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:55.173428 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:55.173449 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:16:55.174196 (+ 747us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:55.174295 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:16:55.174326 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:55.174456 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:55.174673 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 20:16:55.175032 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1665,"thread_start_us":1388,"threads_started":1}
I20260929 20:16:55.180109 28389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:16:55.182070 1484 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1c48f1a06bf41d280b0bb9ee44a6a4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1c48f1a06bf41d280b0bb9ee44a6a4c" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 46725 } }
I20260929 20:16:55.182512 1484 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1c48f1a06bf41d280b0bb9ee44a6a4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:55.182634 1484 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1c48f1a06bf41d280b0bb9ee44a6a4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1c48f1a06bf41d280b0bb9ee44a6a4c, State: Initialized, Role: FOLLOWER
I20260929 20:16:55.183228 1484 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1c48f1a06bf41d280b0bb9ee44a6a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1c48f1a06bf41d280b0bb9ee44a6a4c" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 46725 } }
I20260929 20:16:55.183513 1484 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1c48f1a06bf41d280b0bb9ee44a6a4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:16:55.183638 1484 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1c48f1a06bf41d280b0bb9ee44a6a4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:16:55.183758 1484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1c48f1a06bf41d280b0bb9ee44a6a4c [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:55.186168 1484 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1c48f1a06bf41d280b0bb9ee44a6a4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1c48f1a06bf41d280b0bb9ee44a6a4c" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 46725 } }
I20260929 20:16:55.186611 1484 leader_election.cc:302] T 00000000000000000000000000000000 P b1c48f1a06bf41d280b0bb9ee44a6a4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1c48f1a06bf41d280b0bb9ee44a6a4c; no voters:
I20260929 20:16:55.187165 1484 leader_election.cc:288] T 00000000000000000000000000000000 P b1c48f1a06bf41d280b0bb9ee44a6a4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:16:55.187304 1489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1c48f1a06bf41d280b0bb9ee44a6a4c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:16:55.188043 1489 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1c48f1a06bf41d280b0bb9ee44a6a4c [term 1 LEADER]: Becoming Leader. State: Replica: b1c48f1a06bf41d280b0bb9ee44a6a4c, State: Running, Role: LEADER
I20260929 20:16:55.188530 1489 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1c48f1a06bf41d280b0bb9ee44a6a4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1c48f1a06bf41d280b0bb9ee44a6a4c" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 46725 } }
I20260929 20:16:55.189262 1484 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1c48f1a06bf41d280b0bb9ee44a6a4c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:16:55.191669 1491 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1c48f1a06bf41d280b0bb9ee44a6a4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1c48f1a06bf41d280b0bb9ee44a6a4c. Latest consensus state: current_term: 1 leader_uuid: "b1c48f1a06bf41d280b0bb9ee44a6a4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1c48f1a06bf41d280b0bb9ee44a6a4c" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 46725 } } }
I20260929 20:16:55.192230 1491 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1c48f1a06bf41d280b0bb9ee44a6a4c [sys.catalog]: This master's current role is: LEADER
I20260929 20:16:55.191843 1490 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1c48f1a06bf41d280b0bb9ee44a6a4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1c48f1a06bf41d280b0bb9ee44a6a4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1c48f1a06bf41d280b0bb9ee44a6a4c" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 46725 } } }
I20260929 20:16:55.193575 1490 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1c48f1a06bf41d280b0bb9ee44a6a4c [sys.catalog]: This master's current role is: LEADER
I20260929 20:16:55.194718 1498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:16:55.199949 1498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:16:55.208729 1498 catalog_manager.cc:1347] Generated new cluster ID: 355e328cedd24e3b8d92a16d6bbcf409
I20260929 20:16:55.208873 1498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:16:55.231185 1498 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:16:55.232565 1498 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:16:55.245463 1498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1c48f1a06bf41d280b0bb9ee44a6a4c: Generated new TSK 0
I20260929 20:16:55.246345 1498 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:16:55.306885 1487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:55.259919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39096 (local address 127.27.185.126:46725)
0929 20:16:55.260148 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:55.260157 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:55.260658 (+ 501us) server_negotiation.cc:407] Connection header received
0929 20:16:55.260889 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:55.260893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:55.260957 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:55.261083 (+ 126us) server_negotiation.cc:226] Negotiated authn=JWT
0929 20:16:55.261615 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:55.262341 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:55.264449 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:55.264881 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:55.264977 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:55.265527 (+ 550us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 20:16:55.306306 (+ 40779us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 20:16:55.306637 (+ 331us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39096: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 20:16:55.307058 1487 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39096: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 20:16:55.307746 1513 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:16:55.259908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:46725 (local address 127.0.0.1:39096)
0929 20:16:55.260546 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:55.260568 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:55.260720 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:55.261125 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:55.261130 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:55.261143 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0929 20:16:55.261465 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:55.261478 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:55.262478 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:55.262484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:55.264309 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:55.264325 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:55.265054 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:55.265061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:55.265241 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:55.265284 (+ 43us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 20:16:55.306607 (+ 41323us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 20:16:55.306952 (+ 345us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.185.126:46725: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":321,"threads_started":1}
I20260929 20:16:55.309192 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 1418
I20260929 20:16:55.310382 1479 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:16:55.409963 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 1418
2026-09-29T20:16:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (764 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (415 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (415 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:16:56 dist-test-slave-4cx3 krb5kdc[1528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:16:56 dist-test-slave-4cx3 krb5kdc[1528](info): set up 2 sockets
Sep 29 20:16:56 dist-test-slave-4cx3 krb5kdc[1528](info): commencing operation
krb5kdc: starting...
W20260929 20:16:57.327325 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:16:57 dist-test-slave-4cx3 krb5kdc[1528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:16:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:16:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
I20260929 20:16:57.535912 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:46153
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:37597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:46153
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:57.860891 1541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:57.861270 1541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:57.861449 1541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:57.868752 1541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:16:57.868918 1541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:57.869179 1541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:57.869326 1541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:16:57.869421 1541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:16:57.879541 1541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:37597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:46153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:46153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:57.880959 1541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:57.882325 1541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:57.894240 1549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:57.894217 1546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:57.894300 1547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:57.895162 1541 server_base.cc:1034] running on GCE node
I20260929 20:16:57.896080 1541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:57.897504 1541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:57.898762 1541 hybrid_clock.cc:648] HybridClock initialized: now 1790713017898719 us; error 49 us; skew 500 ppm
I20260929 20:16:57.899132 1541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:57 dist-test-slave-4cx3 krb5kdc[1528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713017, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:57.902303 1541 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:16:57.903795 1541 webserver.cc:460] Webserver started at http://127.27.185.126:37013/ using document root <none> and password file <none>
I20260929 20:16:57.904498 1541 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:57.904659 1541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:57.904996 1541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:57.908036 1541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "6d2c33882bb04f309cdc58599336aa50"
format_stamp: "Formatted at 2026-09-29 20:16:57 on dist-test-slave-4cx3"
server_key: "0e9262bb93a9eff3ba901f421ac41c88"
server_key_iv: "b086a4108a975ff9cb620c27fcbb45dd"
server_key_version: "encryptionkey@0"
I20260929 20:16:57.908823 1541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "6d2c33882bb04f309cdc58599336aa50"
format_stamp: "Formatted at 2026-09-29 20:16:57 on dist-test-slave-4cx3"
server_key: "0e9262bb93a9eff3ba901f421ac41c88"
server_key_iv: "b086a4108a975ff9cb620c27fcbb45dd"
server_key_version: "encryptionkey@0"
I20260929 20:16:57.914040 1541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:16:57.917464 1556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:57.918762 1541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:16:57.918960 1541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "6d2c33882bb04f309cdc58599336aa50"
format_stamp: "Formatted at 2026-09-29 20:16:57 on dist-test-slave-4cx3"
server_key: "0e9262bb93a9eff3ba901f421ac41c88"
server_key_iv: "b086a4108a975ff9cb620c27fcbb45dd"
server_key_version: "encryptionkey@0"
I20260929 20:16:57.919170 1541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:57.938210 1541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:57.942320 1541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:57.942623 1541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:57.961583 1541 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:46153
I20260929 20:16:57.961621 1607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:46153 every 8 connection(s)
I20260929 20:16:57.963045 1541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:16:57.964768 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 1541
I20260929 20:16:57.964999 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:16:57.965660 28389 external_mini_cluster.cc:1428] Setting key 24b84891b983c5d990ba356830ee36a2
I20260929 20:16:57.967449 1608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 20:16:57 dist-test-slave-4cx3 krb5kdc[1528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:57.979562 1608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d2c33882bb04f309cdc58599336aa50: Bootstrap starting.
I20260929 20:16:57.983146 1608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d2c33882bb04f309cdc58599336aa50: Neither blocks nor log segments found. Creating new log.
I20260929 20:16:57.984349 1608 log.cc:824] T 00000000000000000000000000000000 P 6d2c33882bb04f309cdc58599336aa50: Log is configured to *not* fsync() on all Append() calls
I20260929 20:16:57.987304 1608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d2c33882bb04f309cdc58599336aa50: No bootstrap required, opened a new log
I20260929 20:16:57.988477 1611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:57.967769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60638 (local address 127.27.185.126:46153)
0929 20:16:57.968577 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:57.968590 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:57.968631 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:16:57.970278 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:57.970300 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:57.970702 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:57.971151 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:57.972731 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:57.974074 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:57.975302 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:57.976110 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:57.976336 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:57.980621 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:57.980629 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:57.980644 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:57.980687 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:57.984098 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:57.984945 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:57.984953 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:57.984960 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:57.985133 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:57.985557 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:57.985563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:57.985566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:57.986116 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:57.986468 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:57.986891 (+ 423us) server_negotiation.cc:299] Negotiation successful
0929 20:16:57.987364 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":373,"threads_started":1}
I20260929 20:16:57.995759 1608 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d2c33882bb04f309cdc58599336aa50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2c33882bb04f309cdc58599336aa50" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 46153 } }
I20260929 20:16:57.996273 1608 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d2c33882bb04f309cdc58599336aa50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:16:57.996405 1608 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d2c33882bb04f309cdc58599336aa50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d2c33882bb04f309cdc58599336aa50, State: Initialized, Role: FOLLOWER
I20260929 20:16:57.997030 1608 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d2c33882bb04f309cdc58599336aa50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2c33882bb04f309cdc58599336aa50" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 46153 } }
I20260929 20:16:57.997270 1608 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d2c33882bb04f309cdc58599336aa50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:16:57.997422 1608 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d2c33882bb04f309cdc58599336aa50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:16:57.997608 1608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d2c33882bb04f309cdc58599336aa50 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:16:58.000114 1608 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d2c33882bb04f309cdc58599336aa50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2c33882bb04f309cdc58599336aa50" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 46153 } }
I20260929 20:16:58.000635 1608 leader_election.cc:302] T 00000000000000000000000000000000 P 6d2c33882bb04f309cdc58599336aa50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d2c33882bb04f309cdc58599336aa50; no voters:
I20260929 20:16:58.001216 1608 leader_election.cc:288] T 00000000000000000000000000000000 P 6d2c33882bb04f309cdc58599336aa50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:16:58.001343 1613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d2c33882bb04f309cdc58599336aa50 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:16:58.002149 1613 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d2c33882bb04f309cdc58599336aa50 [term 1 LEADER]: Becoming Leader. State: Replica: 6d2c33882bb04f309cdc58599336aa50, State: Running, Role: LEADER
I20260929 20:16:58.002674 1613 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d2c33882bb04f309cdc58599336aa50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2c33882bb04f309cdc58599336aa50" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 46153 } }
I20260929 20:16:58.003576 1608 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d2c33882bb04f309cdc58599336aa50 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:16:58.005334 1614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d2c33882bb04f309cdc58599336aa50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d2c33882bb04f309cdc58599336aa50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2c33882bb04f309cdc58599336aa50" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 46153 } } }
I20260929 20:16:58.005755 1614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d2c33882bb04f309cdc58599336aa50 [sys.catalog]: This master's current role is: LEADER
I20260929 20:16:58.005811 1615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d2c33882bb04f309cdc58599336aa50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d2c33882bb04f309cdc58599336aa50. Latest consensus state: current_term: 1 leader_uuid: "6d2c33882bb04f309cdc58599336aa50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2c33882bb04f309cdc58599336aa50" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 46153 } } }
I20260929 20:16:58.006093 1615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d2c33882bb04f309cdc58599336aa50 [sys.catalog]: This master's current role is: LEADER
I20260929 20:16:58.012099 1618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:16:58.017297 1618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:16:58.027129 1618 catalog_manager.cc:1347] Generated new cluster ID: 58b99aaeb08b415588d1df4e2c3c0bda
I20260929 20:16:58.027289 1618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:16:58.041076 1618 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:16:58.042979 1618 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:16:58.057937 1618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d2c33882bb04f309cdc58599336aa50: Generated new TSK 0
I20260929 20:16:58.058825 1618 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
I20260929 20:16:58.115013 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:0
--local_ip_for_outbound_sockets=127.27.185.65
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:46153
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:37597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:58.446463 1634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:58.446744 1634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:58.446902 1634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:58.454227 1634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:58.454375 1634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:58.454568 1634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:16:58.464407 1634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:37597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:46153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:58.465731 1634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:58.467159 1634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:58.480460 1640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:58.480480 1639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:58.481278 1642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:58.481334 1634 server_base.cc:1034] running on GCE node
I20260929 20:16:58.482151 1634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:58.483103 1634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:58.484330 1634 hybrid_clock.cc:648] HybridClock initialized: now 1790713018484288 us; error 57 us; skew 500 ppm
I20260929 20:16:58.484647 1634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:58 dist-test-slave-4cx3 krb5kdc[1528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713018, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:58.487709 1634 init.cc:377] Logged in from keytab as kudu/127.27.185.65@KRBTEST.COM (short username kudu)
I20260929 20:16:58.489239 1634 webserver.cc:460] Webserver started at http://127.27.185.65:36837/ using document root <none> and password file <none>
I20260929 20:16:58.489961 1634 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:58.490115 1634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:58.490455 1634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:58.493413 1634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-0/data/instance:
uuid: "56612bea35e74c8381f65768abb320c3"
format_stamp: "Formatted at 2026-09-29 20:16:58 on dist-test-slave-4cx3"
server_key: "32e541e242eb6ef940fed0e2df61498d"
server_key_iv: "5c8e91571e2db19ee8dbd86de064e557"
server_key_version: "encryptionkey@0"
I20260929 20:16:58.494184 1634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance:
uuid: "56612bea35e74c8381f65768abb320c3"
format_stamp: "Formatted at 2026-09-29 20:16:58 on dist-test-slave-4cx3"
server_key: "32e541e242eb6ef940fed0e2df61498d"
server_key_iv: "5c8e91571e2db19ee8dbd86de064e557"
server_key_version: "encryptionkey@0"
I20260929 20:16:58.499295 1634 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 20:16:58.502697 1649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:58.504087 1634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:16:58.504245 1634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "56612bea35e74c8381f65768abb320c3"
format_stamp: "Formatted at 2026-09-29 20:16:58 on dist-test-slave-4cx3"
server_key: "32e541e242eb6ef940fed0e2df61498d"
server_key_iv: "5c8e91571e2db19ee8dbd86de064e557"
server_key_version: "encryptionkey@0"
I20260929 20:16:58.504453 1634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:58.534260 1634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:58.538966 1634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:58.539273 1634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:58.540261 1634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:16:58.541596 1634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:16:58.541667 1634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:58.541817 1634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:16:58.541904 1634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:58.573694 1634 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:45989
I20260929 20:16:58.573729 1761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:45989 every 8 connection(s)
I20260929 20:16:58.575080 1634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:16:58.578419 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 1634
I20260929 20:16:58.578661 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance
I20260929 20:16:58.579588 28389 external_mini_cluster.cc:1428] Setting key 18cf6bc868c144d36ad4fac8f54b63a7
Sep 29 20:16:58 dist-test-slave-4cx3 krb5kdc[1528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713018, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:58.597975 1764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:58.577919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:33173 (local address 127.27.185.126:46153)
0929 20:16:58.578404 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:58.578410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:58.581273 (+ 2863us) server_negotiation.cc:407] Connection header received
0929 20:16:58.581823 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:58.581829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:58.581912 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:58.582045 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:58.583605 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:58.584753 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:58.586349 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:58.587074 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:58.587226 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:58.590893 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:58.590903 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:58.590910 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:58.590964 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:58.594535 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:58.595305 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:58.595313 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:58.595319 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:58.595527 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:58.595952 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:58.595960 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:58.595966 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:58.596356 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:58.596522 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:58.597229 (+ 707us) server_negotiation.cc:299] Negotiation successful
0929 20:16:58.597503 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":267,"threads_started":1}
I20260929 20:16:58.598799 1765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:58.578675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:46153 (local address 127.27.185.65:33173)
0929 20:16:58.579576 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:58.579622 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:58.581555 (+ 1933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:58.582246 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:58.582254 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:58.582786 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:58.583373 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:58.583452 (+ 79us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:58.584917 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:58.584920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:58.586194 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:58.586209 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:58.587288 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:58.587293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:58.587565 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:58.588271 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:58.588293 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:58.590652 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:58.594693 (+ 4041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:58.594700 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:58.594715 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:58.595163 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:58.595637 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:58.595640 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:58.595643 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:58.595821 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:58.596480 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:58.596489 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:58.596803 (+ 314us) client_negotiation.cc:769] Sending connection context
0929 20:16:58.597046 (+ 243us) client_negotiation.cc:240] Negotiation successful
0929 20:16:58.597538 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":362,"threads_started":1}
I20260929 20:16:58.600574 1762 heartbeater.cc:347] Connected to a master server at 127.27.185.126:46153
I20260929 20:16:58.600888 1762 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.27.185.66@KRBTEST.COM; defaulting to no policy
I20260929 20:16:58.601662 1762 heartbeater.cc:511] Master 127.27.185.126:46153 requested a full tablet report, sending...
I20260929 20:16:58.603987 1573 ts_manager.cc:194] Registered new tserver with Master: 56612bea35e74c8381f65768abb320c3 (127.27.185.65:45989)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.66@KRBTEST.COM" created.
I20260929 20:16:58.605593 1573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.65@KRBTEST.COM'} at 127.27.185.65:33173
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
I20260929 20:16:58.630968 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:0
--local_ip_for_outbound_sockets=127.27.185.66
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:46153
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:37597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:58.961632 1768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:58.961905 1768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:58.962069 1768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:58.969616 1768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:58.969730 1768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:58.969913 1768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:16:58.980170 1768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:37597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:46153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:58.981514 1768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:58.982851 1768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:58.996610 1773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:58.996632 1774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:58.996764 1776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:58.997249 1768 server_base.cc:1034] running on GCE node
I20260929 20:16:58.997917 1768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:58.998888 1768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:59.000159 1768 hybrid_clock.cc:648] HybridClock initialized: now 1790713019000086 us; error 50 us; skew 500 ppm
I20260929 20:16:59.000618 1768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:58 dist-test-slave-4cx3 krb5kdc[1528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713018, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:59.003862 1768 init.cc:377] Logged in from keytab as kudu/127.27.185.66@KRBTEST.COM (short username kudu)
I20260929 20:16:59.005432 1768 webserver.cc:460] Webserver started at http://127.27.185.66:45525/ using document root <none> and password file <none>
I20260929 20:16:59.006085 1768 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:59.006186 1768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:59.006443 1768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:59.009570 1768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-1/data/instance:
uuid: "75c5a7fe27e5433cba9ce33478b351c2"
format_stamp: "Formatted at 2026-09-29 20:16:59 on dist-test-slave-4cx3"
server_key: "b2fec3c8acd6e4fb23a2964fe58dcead"
server_key_iv: "ef8c9a558127341b2853d9a512a03dc6"
server_key_version: "encryptionkey@0"
I20260929 20:16:59.010349 1768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance:
uuid: "75c5a7fe27e5433cba9ce33478b351c2"
format_stamp: "Formatted at 2026-09-29 20:16:59 on dist-test-slave-4cx3"
server_key: "b2fec3c8acd6e4fb23a2964fe58dcead"
server_key_iv: "ef8c9a558127341b2853d9a512a03dc6"
server_key_version: "encryptionkey@0"
I20260929 20:16:59.015609 1768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 20:16:59.019268 1783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:59.020678 1768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:16:59.020814 1768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "75c5a7fe27e5433cba9ce33478b351c2"
format_stamp: "Formatted at 2026-09-29 20:16:59 on dist-test-slave-4cx3"
server_key: "b2fec3c8acd6e4fb23a2964fe58dcead"
server_key_iv: "ef8c9a558127341b2853d9a512a03dc6"
server_key_version: "encryptionkey@0"
I20260929 20:16:59.020991 1768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:59.037817 1768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:59.042313 1768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:59.042577 1768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:59.043573 1768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:16:59.044872 1768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:16:59.044945 1768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:59.045030 1768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:16:59.045076 1768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:59.079739 1768 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:40375
I20260929 20:16:59.079834 1895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:40375 every 8 connection(s)
I20260929 20:16:59.081156 1768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:16:59.090211 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 1768
I20260929 20:16:59.090430 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance
I20260929 20:16:59.090952 28389 external_mini_cluster.cc:1428] Setting key 98d4e9e286fcced10988bc65cfa7e487
Sep 29 20:16:59 dist-test-slave-4cx3 krb5kdc[1528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713018, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:59.102909 1764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:59.084453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:60859 (local address 127.27.185.126:46153)
0929 20:16:59.084650 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:59.084655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:59.085975 (+ 1320us) server_negotiation.cc:407] Connection header received
0929 20:16:59.087924 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:59.087930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:59.088024 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:59.088139 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:59.089731 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:59.090908 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:59.092972 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:59.093396 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:59.093542 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:59.097155 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:59.097160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:59.097163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:59.097195 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:59.099886 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:59.100629 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:59.100638 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:59.100641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:59.100762 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:59.101205 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:59.101215 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:59.101218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:59.101478 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:59.101634 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:59.102328 (+ 694us) server_negotiation.cc:299] Negotiation successful
0929 20:16:59.102571 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 20:16:59.104102 1898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:59.085054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:46153 (local address 127.27.185.66:60859)
0929 20:16:59.085801 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:59.085842 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:59.087633 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:59.088386 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:59.088395 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:59.088946 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:59.089548 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:59.089564 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:59.091133 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:59.091138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:59.092805 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:59.092820 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:59.093635 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:59.093642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:59.093887 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:59.094639 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:59.094663 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:59.096913 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:59.100044 (+ 3131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:59.100053 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:59.100071 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:59.100498 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:59.100876 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:59.100882 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:59.100885 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:59.101084 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:59.101606 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:59.101614 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:59.101965 (+ 351us) client_negotiation.cc:769] Sending connection context
0929 20:16:59.102233 (+ 268us) client_negotiation.cc:240] Negotiation successful
0929 20:16:59.102795 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":303,"threads_started":1}
I20260929 20:16:59.105695 1896 heartbeater.cc:347] Connected to a master server at 127.27.185.126:46153
I20260929 20:16:59.106046 1896 heartbeater.cc:464] Registering TS with master...
I20260929 20:16:59.107035 1896 heartbeater.cc:511] Master 127.27.185.126:46153 requested a full tablet report, sending...
I20260929 20:16:59.108906 1573 ts_manager.cc:194] Registered new tserver with Master: 75c5a7fe27e5433cba9ce33478b351c2 (127.27.185.66:40375)
I20260929 20:16:59.110204 1573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.66@KRBTEST.COM'} at 127.27.185.66:60859
WARNING: no policy specified for kudu/127.27.185.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
I20260929 20:16:59.139461 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:0
--local_ip_for_outbound_sockets=127.27.185.67
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:46153
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:37597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:16:59.466670 1901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:16:59.466961 1901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:16:59.467133 1901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:16:59.474467 1901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:16:59.474583 1901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:16:59.474785 1901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:16:59.484519 1901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:37597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:46153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:16:59.485842 1901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:16:59.487157 1901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:16:59.500653 1909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:59.500713 1907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:16:59.500677 1906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:16:59.501559 1901 server_base.cc:1034] running on GCE node
I20260929 20:16:59.502185 1901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:16:59.503144 1901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:16:59.504395 1901 hybrid_clock.cc:648] HybridClock initialized: now 1790713019504263 us; error 136 us; skew 500 ppm
I20260929 20:16:59.504704 1901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:16:59 dist-test-slave-4cx3 krb5kdc[1528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713019, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:59.507993 1901 init.cc:377] Logged in from keytab as kudu/127.27.185.67@KRBTEST.COM (short username kudu)
I20260929 20:16:59.509609 1901 webserver.cc:460] Webserver started at http://127.27.185.67:46855/ using document root <none> and password file <none>
I20260929 20:16:59.510365 1901 fs_manager.cc:359] Metadata directory not provided
I20260929 20:16:59.510510 1901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:16:59.510838 1901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:16:59.513880 1901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-2/data/instance:
uuid: "431abf693a054db09cc387c9a5db5f15"
format_stamp: "Formatted at 2026-09-29 20:16:59 on dist-test-slave-4cx3"
server_key: "82629972c263bac52e629e178c0bf63d"
server_key_iv: "4a69df6b0202dbbc9b798b1a6b3a9887"
server_key_version: "encryptionkey@0"
I20260929 20:16:59.514699 1901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance:
uuid: "431abf693a054db09cc387c9a5db5f15"
format_stamp: "Formatted at 2026-09-29 20:16:59 on dist-test-slave-4cx3"
server_key: "82629972c263bac52e629e178c0bf63d"
server_key_iv: "4a69df6b0202dbbc9b798b1a6b3a9887"
server_key_version: "encryptionkey@0"
I20260929 20:16:59.520107 1901 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260929 20:16:59.523911 1916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:59.525487 1901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:16:59.525645 1901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "431abf693a054db09cc387c9a5db5f15"
format_stamp: "Formatted at 2026-09-29 20:16:59 on dist-test-slave-4cx3"
server_key: "82629972c263bac52e629e178c0bf63d"
server_key_iv: "4a69df6b0202dbbc9b798b1a6b3a9887"
server_key_version: "encryptionkey@0"
I20260929 20:16:59.525856 1901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:16:59.538602 1901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:16:59.542716 1901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:16:59.543008 1901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:16:59.544051 1901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:16:59.545370 1901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:16:59.545450 1901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:59.545598 1901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:16:59.545658 1901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:16:59.579704 1901 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:33467
I20260929 20:16:59.579722 2028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:33467 every 8 connection(s)
I20260929 20:16:59.581365 1901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:16:59.589546 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 1901
I20260929 20:16:59.589767 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance
I20260929 20:16:59.590315 28389 external_mini_cluster.cc:1428] Setting key a848b358e84990ef0448b43da621dc17
Sep 29 20:16:59 dist-test-slave-4cx3 krb5kdc[1528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713019, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:16:59.602258 1764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:59.584207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:52981 (local address 127.27.185.126:46153)
0929 20:16:59.584429 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:59.584434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:59.585717 (+ 1283us) server_negotiation.cc:407] Connection header received
0929 20:16:59.587650 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:59.587655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:59.587745 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:59.587844 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:59.589476 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:59.590657 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:59.592745 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:59.593385 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:59.593547 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:59.596969 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:59.596974 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:59.596977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:59.597012 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:59.599638 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:59.600342 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:59.600346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:59.600348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:59.600469 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:59.600856 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:59.600859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:59.600861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:59.601103 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:59.601208 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:59.601855 (+ 647us) server_negotiation.cc:299] Negotiation successful
0929 20:16:59.602013 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 20:16:59.603300 2031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:59.584735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:46153 (local address 127.27.185.67:52981)
0929 20:16:59.585539 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0929 20:16:59.585585 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:16:59.587265 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:16:59.588080 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:16:59.588089 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:16:59.588659 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:16:59.589266 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:59.589298 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:59.590819 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:59.590823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:59.592589 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:16:59.592605 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:16:59.593613 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:16:59.593620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:16:59.593820 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:59.594517 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:16:59.594541 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:16:59.596776 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:16:59.599768 (+ 2992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:59.599775 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:59.599789 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:59.600235 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:59.600574 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:16:59.600577 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:16:59.600579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:16:59.600761 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:16:59.601202 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:16:59.601208 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:16:59.601542 (+ 334us) client_negotiation.cc:769] Sending connection context
0929 20:16:59.601762 (+ 220us) client_negotiation.cc:240] Negotiation successful
0929 20:16:59.602152 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":329,"threads_started":1}
I20260929 20:16:59.604912 2029 heartbeater.cc:347] Connected to a master server at 127.27.185.126:46153
I20260929 20:16:59.605295 2029 heartbeater.cc:464] Registering TS with master...
I20260929 20:16:59.606194 2029 heartbeater.cc:511] Master 127.27.185.126:46153 requested a full tablet report, sending...
I20260929 20:16:59.607630 1573 ts_manager.cc:194] Registered new tserver with Master: 431abf693a054db09cc387c9a5db5f15 (127.27.185.67:33467)
I20260929 20:16:59.608502 1573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.67@KRBTEST.COM'} at 127.27.185.67:52981
I20260929 20:16:59.608796 1762 heartbeater.cc:503] Master 127.27.185.126:46153 was elected leader, sending a full tablet report...
I20260929 20:16:59.618208 28389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/test-admin.keytab.
Sep 29 20:16:59 dist-test-slave-4cx3 krb5kdc[1528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:59.645246 28389 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 20:16:59.650665 2036 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:16:59.647958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43830 (local address 127.27.185.65:45989)
0929 20:16:59.648651 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:59.648660 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:59.648694 (+ 34us) server_negotiation.cc:407] Connection header received
0929 20:16:59.648758 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:59.648764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:59.649023 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:59.649206 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:59.650241 (+ 1035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43830: BlockingRecv error: recv got EOF from 127.0.0.1:43830 (error 108)
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":324,"threads_started":1}
W20260929 20:16:59.651153 28389 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/krb5cc' of type FILE
Sep 29 20:16:59 dist-test-slave-4cx3 krb5kdc[1528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:59.652770 28389 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 20:16:59 dist-test-slave-4cx3 krb5kdc[1528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:59.654109 28389 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 20:16:59 dist-test-slave-4cx3 krb5kdc[1528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.65@KRBTEST.COM
I20260929 20:16:59.670580 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:59.655695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43846 (local address 127.27.185.65:45989)
0929 20:16:59.655857 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:59.655861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:59.656213 (+ 352us) server_negotiation.cc:407] Connection header received
0929 20:16:59.656344 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:59.656348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:59.656409 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:59.656494 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:59.657436 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:59.658387 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:59.659657 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:59.660221 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:59.660330 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:59.663961 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:59.663973 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:59.663994 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:59.664033 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:59.668045 (+ 4012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:59.668684 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:59.668691 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:59.668694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:59.668818 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:59.669150 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:59.669153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:59.669155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:59.669524 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:59.669738 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:59.669996 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 20:16:59.670263 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 20:16:59.676234 2036 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:16:59.674425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43860 (local address 127.27.185.65:45989)
0929 20:16:59.674638 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:59.674645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:59.674895 (+ 250us) server_negotiation.cc:407] Connection header received
0929 20:16:59.675090 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:59.675094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:59.675150 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:59.675222 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:59.675995 (+ 773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43860: BlockingRecv error: recv got EOF from 127.0.0.1:43860 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 20:16:59.676349 28389 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/krb5cc' of type FILE
Sep 29 20:16:59 dist-test-slave-4cx3 krb5kdc[1528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:59.677980 28389 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 20:16:59 dist-test-slave-4cx3 krb5kdc[1528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:59.679317 28389 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 20:16:59 dist-test-slave-4cx3 krb5kdc[1528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.65@KRBTEST.COM
I20260929 20:16:59.693703 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:59.680664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43864 (local address 127.27.185.65:45989)
0929 20:16:59.680860 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:59.680867 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:59.681129 (+ 262us) server_negotiation.cc:407] Connection header received
0929 20:16:59.681324 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:59.681328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:59.681386 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:59.681502 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:59.682468 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:59.683147 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:59.684391 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:59.684814 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:59.684947 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:59.688386 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:59.688391 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:59.688394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:59.688431 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:59.691068 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:59.691997 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:59.692001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:59.692003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:59.692130 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:59.692520 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:59.692523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:59.692525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:59.692778 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:59.692968 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:59.693216 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 20:16:59.693423 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 20:16:59.697790 2036 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:16:59.696371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43880 (local address 127.27.185.65:45989)
0929 20:16:59.696574 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:59.696578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:59.696678 (+ 100us) server_negotiation.cc:407] Connection header received
0929 20:16:59.696827 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:59.696830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:59.696894 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:59.696967 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:59.697570 (+ 603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43880: BlockingRecv error: recv got EOF from 127.0.0.1:43880 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 20:16:59.697965 28389 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790712966917858-28389-0/krb5kdc/krb5cc' of type FILE
Sep 29 20:16:59 dist-test-slave-4cx3 krb5kdc[1528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:59.699649 28389 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 20:16:59 dist-test-slave-4cx3 krb5kdc[1528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:16:59.700994 28389 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 20:16:59 dist-test-slave-4cx3 krb5kdc[1528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.65@KRBTEST.COM
I20260929 20:16:59.714833 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:59.702267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43888 (local address 127.27.185.65:45989)
0929 20:16:59.702502 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:59.702508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:59.702707 (+ 199us) server_negotiation.cc:407] Connection header received
0929 20:16:59.702833 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:59.702837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:59.702897 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:59.702976 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:59.704064 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:59.704765 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:59.705997 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:59.706398 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:59.706495 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:59.710177 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:59.710182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:59.710185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:59.710217 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:59.712626 (+ 2409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:59.713203 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:59.713207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:59.713210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:59.713328 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:59.713626 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:59.713629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:59.713631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:59.713894 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:59.714120 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:59.714297 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 20:16:59.714456 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:16:59.716590 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 1634
I20260929 20:16:59.729671 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:59.717410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43894 (local address 127.27.185.65:45989)
0929 20:16:59.717615 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:59.717622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:59.717843 (+ 221us) server_negotiation.cc:407] Connection header received
0929 20:16:59.718034 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:59.718044 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:59.718099 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:59.718196 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:59.719135 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:59.719884 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:59.721125 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:59.721558 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:59.721708 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:59.724132 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:59.724137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:59.724140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:59.724172 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:59.726933 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:59.727663 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:59.727670 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:59.727674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:59.727810 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:59.728189 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:59.728194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:59.728197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:59.728458 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:59.728726 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:59.729046 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 20:16:59.729258 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:16:59.730357 1757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:16:59.814859 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 1634
I20260929 20:16:59.831321 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 1768
Sep 29 20:16:59 dist-test-slave-4cx3 krb5kdc[1528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.66@KRBTEST.COM
I20260929 20:16:59.847249 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:59.832329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53348 (local address 127.27.185.66:40375)
0929 20:16:59.832852 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:59.832859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:59.832885 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:16:59.832945 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:59.832951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:59.833169 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:59.833295 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:59.834441 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:59.835470 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:59.836766 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:59.837309 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:59.837408 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:59.841623 (+ 4215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:59.841635 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:59.841648 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:59.841681 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:59.844609 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:59.845314 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:59.845320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:59.845323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:59.845437 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:59.845805 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:59.845808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:59.845810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:59.846141 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:59.846376 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:59.846697 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 20:16:59.846905 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":273,"threads_started":1}
I20260929 20:16:59.848639 1891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:16:59.930639 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 1768
I20260929 20:16:59.946153 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 1901
Sep 29 20:16:59 dist-test-slave-4cx3 krb5kdc[1528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.67@KRBTEST.COM
I20260929 20:16:59.961761 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:16:59.947163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52074 (local address 127.27.185.67:33467)
0929 20:16:59.947834 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0929 20:16:59.947841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:16:59.947871 (+ 30us) server_negotiation.cc:407] Connection header received
0929 20:16:59.947930 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:16:59.947937 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:16:59.948181 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:16:59.948311 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:16:59.949471 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:59.950499 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:59.951711 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:16:59.952262 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:16:59.952358 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:16:59.956271 (+ 3913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:16:59.956282 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:16:59.956296 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:16:59.956329 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:16:59.959006 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:59.959714 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:59.959721 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:59.959723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:59.959845 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:16:59.960173 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:16:59.960176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:16:59.960178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:16:59.960540 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:16:59.960769 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 20:16:59.961027 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 20:16:59.961355 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":367,"threads_started":1}
I20260929 20:16:59.963181 2024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:00.048332 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 1901
I20260929 20:17:00.064718 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 1541
I20260929 20:17:00.066138 1603 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:00.171012 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 1541
2026-09-29T20:17:00Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3935 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:17:00 dist-test-slave-4cx3 krb5kdc[2056](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:17:00 dist-test-slave-4cx3 krb5kdc[2056](info): set up 2 sockets
Sep 29 20:17:00 dist-test-slave-4cx3 krb5kdc[2056](info): commencing operation
krb5kdc: starting...
W20260929 20:17:01.275267 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:17:01 dist-test-slave-4cx3 krb5kdc[2056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:17:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:17:01Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.126.keytab.
Entry for principal oryx/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.126.keytab.
I20260929 20:17:01.491257 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:36737
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.126.keytab
--principal=oryx/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:40591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:36737
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:17:01.827646 2069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:01.827955 2069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:01.828049 2069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:01.835534 2069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:17:01.835659 2069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:01.835707 2069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:01.835757 2069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:17:01.835803 2069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:17:01.846247 2069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:40591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:36737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.126.keytab
--principal=oryx/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:36737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:01.847653 2069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:01.848948 2069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:01.861755 2077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:01.861893 2075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:01.862063 2074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:01.864113 2069 server_base.cc:1034] running on GCE node
I20260929 20:17:01.864989 2069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:01.866510 2069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:01.867751 2069 hybrid_clock.cc:648] HybridClock initialized: now 1790713021867695 us; error 76 us; skew 500 ppm
I20260929 20:17:01.868131 2069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:17:01 dist-test-slave-4cx3 krb5kdc[2056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713021, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:17:01.871582 2069 init.cc:377] Logged in from keytab as oryx/127.27.185.126@KRBTEST.COM (short username oryx)
I20260929 20:17:01.873276 2069 webserver.cc:460] Webserver started at http://127.27.185.126:42661/ using document root <none> and password file <none>
I20260929 20:17:01.874009 2069 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:01.874110 2069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:01.874377 2069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:17:01.877579 2069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "3a60def670884f8886ada0b1616e4159"
format_stamp: "Formatted at 2026-09-29 20:17:01 on dist-test-slave-4cx3"
server_key: "301b944910b41062c9156e62ae4df1c1"
server_key_iv: "80fafcd87e7b38e3d29bdd6e16fb20cf"
server_key_version: "encryptionkey@0"
I20260929 20:17:01.878364 2069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "3a60def670884f8886ada0b1616e4159"
format_stamp: "Formatted at 2026-09-29 20:17:01 on dist-test-slave-4cx3"
server_key: "301b944910b41062c9156e62ae4df1c1"
server_key_iv: "80fafcd87e7b38e3d29bdd6e16fb20cf"
server_key_version: "encryptionkey@0"
I20260929 20:17:01.883848 2069 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:17:01.887559 2084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:01.888963 2069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:17:01.889154 2069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "3a60def670884f8886ada0b1616e4159"
format_stamp: "Formatted at 2026-09-29 20:17:01 on dist-test-slave-4cx3"
server_key: "301b944910b41062c9156e62ae4df1c1"
server_key_iv: "80fafcd87e7b38e3d29bdd6e16fb20cf"
server_key_version: "encryptionkey@0"
I20260929 20:17:01.889380 2069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:01.931360 2069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:01.936014 2069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:01.936347 2069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:01.955580 2069 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:36737
I20260929 20:17:01.955611 2135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:36737 every 8 connection(s)
I20260929 20:17:01.956986 2069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:17:01.960912 2136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:01.963054 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 2069
I20260929 20:17:01.963312 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:17:01.963971 28389 external_mini_cluster.cc:1428] Setting key 1a31be633a9e3a48e33f44488467dbeb
I20260929 20:17:01.973169 2136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159: Bootstrap starting.
I20260929 20:17:01.977859 2136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:01.979645 2136 log.cc:824] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159: Log is configured to *not* fsync() on all Append() calls
Sep 29 20:17:01 dist-test-slave-4cx3 krb5kdc[2056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM
I20260929 20:17:01.988783 2136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159: No bootstrap required, opened a new log
I20260929 20:17:01.997190 2139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:01.966129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37964 (local address 127.27.185.126:36737)
0929 20:17:01.966972 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:01.966983 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:01.967025 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:17:01.968742 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:01.968768 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:01.969258 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:01.969638 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:01.971703 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:01.973059 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:01.975028 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:01.975856 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:01.976094 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:01.981733 (+ 5639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:01.981750 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:01.981776 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:01.981846 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:01.989964 (+ 8118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:01.991133 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:01.991146 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:01.991159 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:01.991446 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:01.992019 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:01.992027 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:01.992032 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:01.992956 (+ 924us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:01.993304 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:01.994834 (+ 1530us) server_negotiation.cc:299] Negotiation successful
0929 20:17:01.995453 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":406,"threads_started":1}
I20260929 20:17:02.003473 2136 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } }
I20260929 20:17:02.003968 2136 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:02.004096 2136 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a60def670884f8886ada0b1616e4159, State: Initialized, Role: FOLLOWER
I20260929 20:17:02.004806 2136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } }
I20260929 20:17:02.005072 2136 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:17:02.005223 2136 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:17:02.005407 2136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:17:02.008394 2136 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } }
I20260929 20:17:02.008925 2136 leader_election.cc:302] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a60def670884f8886ada0b1616e4159; no voters:
I20260929 20:17:02.009578 2136 leader_election.cc:288] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:17:02.009739 2141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:17:02.010661 2141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [term 1 LEADER]: Becoming Leader. State: Replica: 3a60def670884f8886ada0b1616e4159, State: Running, Role: LEADER
I20260929 20:17:02.011305 2141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } }
I20260929 20:17:02.011901 2136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:17:02.014441 2142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a60def670884f8886ada0b1616e4159" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } }
I20260929 20:17:02.014798 2142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [sys.catalog]: This master's current role is: LEADER
I20260929 20:17:02.014554 2143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a60def670884f8886ada0b1616e4159. Latest consensus state: current_term: 1 leader_uuid: "3a60def670884f8886ada0b1616e4159" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } }
I20260929 20:17:02.014998 2143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [sys.catalog]: This master's current role is: LEADER
I20260929 20:17:02.020453 2147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:17:02.025801 2147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:17:02.034348 2147 catalog_manager.cc:1347] Generated new cluster ID: 09b63d71bcc342fcaf7a45474598022f
I20260929 20:17:02.034477 2147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:17:02.066052 2147 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:17:02.068152 2147 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:17:02.080652 2147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159: Generated new TSK 0
I20260929 20:17:02.081743 2147 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.65.keytab.
Entry for principal oryx/127.27.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.65.keytab.
I20260929 20:17:02.163312 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:0
--local_ip_for_outbound_sockets=127.27.185.65
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:36737
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.65.keytab
--principal=oryx/127.27.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:40591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:17:02.503476 2162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:02.503829 2162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:02.503998 2162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:02.513823 2162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:02.514076 2162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:02.514307 2162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:17:02.525024 2162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:40591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.65.keytab
--principal=oryx/127.27.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:36737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:02.526335 2162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:02.527717 2162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:02.540912 2168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:02.540931 2170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:02.540859 2167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:02.541666 2162 server_base.cc:1034] running on GCE node
I20260929 20:17:02.542310 2162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:02.543284 2162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:02.544505 2162 hybrid_clock.cc:648] HybridClock initialized: now 1790713022544472 us; error 49 us; skew 500 ppm
I20260929 20:17:02.544826 2162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:17:02 dist-test-slave-4cx3 krb5kdc[2056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713022, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:17:02.547967 2162 init.cc:377] Logged in from keytab as oryx/127.27.185.65@KRBTEST.COM (short username oryx)
I20260929 20:17:02.549400 2162 webserver.cc:460] Webserver started at http://127.27.185.65:43195/ using document root <none> and password file <none>
I20260929 20:17:02.550155 2162 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:02.550261 2162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:02.550506 2162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:17:02.553491 2162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/data/instance:
uuid: "eaa3fccfe54b400e80a3f4c5c33a1baf"
format_stamp: "Formatted at 2026-09-29 20:17:02 on dist-test-slave-4cx3"
server_key: "bfe614719920366408765a5601d64f33"
server_key_iv: "d209c48deea247af2ebd40a555b40a57"
server_key_version: "encryptionkey@0"
I20260929 20:17:02.554261 2162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance:
uuid: "eaa3fccfe54b400e80a3f4c5c33a1baf"
format_stamp: "Formatted at 2026-09-29 20:17:02 on dist-test-slave-4cx3"
server_key: "bfe614719920366408765a5601d64f33"
server_key_iv: "d209c48deea247af2ebd40a555b40a57"
server_key_version: "encryptionkey@0"
I20260929 20:17:02.559495 2162 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 20:17:02.562968 2177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:02.564508 2162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 20:17:02.564687 2162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "eaa3fccfe54b400e80a3f4c5c33a1baf"
format_stamp: "Formatted at 2026-09-29 20:17:02 on dist-test-slave-4cx3"
server_key: "bfe614719920366408765a5601d64f33"
server_key_iv: "d209c48deea247af2ebd40a555b40a57"
server_key_version: "encryptionkey@0"
I20260929 20:17:02.564874 2162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:02.608470 2162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:02.613005 2162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:02.613332 2162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:02.614394 2162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:17:02.615897 2162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:17:02.615980 2162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:02.616107 2162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:17:02.616173 2162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:02.652096 2162 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:42717
I20260929 20:17:02.652139 2289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:42717 every 8 connection(s)
I20260929 20:17:02.653465 2162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:17:02.655678 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 2162
I20260929 20:17:02.655937 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance
I20260929 20:17:02.656723 28389 external_mini_cluster.cc:1428] Setting key 95cc3e5bb30a1c4e225c707c2bfc6519
Sep 29 20:17:02 dist-test-slave-4cx3 krb5kdc[2056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713022, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM
I20260929 20:17:02.676390 2292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:02.656494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:52685 (local address 127.27.185.126:36737)
0929 20:17:02.657372 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:02.657381 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:02.658221 (+ 840us) server_negotiation.cc:407] Connection header received
0929 20:17:02.660320 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:02.660329 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:02.660432 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:02.660554 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:02.662077 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:02.663258 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:02.665087 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:02.665490 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:02.665590 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:02.670241 (+ 4651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:02.670246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:02.670249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:02.670284 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:02.673020 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:02.673815 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:02.673826 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:02.673829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:02.673954 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:02.674447 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:02.674452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:02.674454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:02.674738 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:02.674913 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:02.675762 (+ 849us) server_negotiation.cc:299] Negotiation successful
0929 20:17:02.676014 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":614,"threads_started":1}
I20260929 20:17:02.677852 2293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:02.657059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.27.185.65:52685)
0929 20:17:02.658038 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:02.658092 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:02.660043 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:02.660767 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:02.660775 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:02.661315 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:02.661901 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:02.661919 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:02.663495 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:02.663500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:02.664894 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:02.664913 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:02.665737 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:02.665746 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:02.666109 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:02.666895 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:02.666933 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:02.669981 (+ 3048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:02.673184 (+ 3203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:02.673194 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:02.673213 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:02.673673 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:02.674075 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:02.674079 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:02.674083 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:02.674304 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:02.674885 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:02.674894 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:02.675287 (+ 393us) client_negotiation.cc:769] Sending connection context
0929 20:17:02.675653 (+ 366us) client_negotiation.cc:240] Negotiation successful
0929 20:17:02.676265 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":424,"threads_started":1}
WARNING: no policy specified for oryx/127.27.185.66@KRBTEST.COM; defaulting to no policy
I20260929 20:17:02.680399 2290 heartbeater.cc:347] Connected to a master server at 127.27.185.126:36737
I20260929 20:17:02.680745 2290 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.185.66@KRBTEST.COM" created.
I20260929 20:17:02.681684 2290 heartbeater.cc:511] Master 127.27.185.126:36737 requested a full tablet report, sending...
I20260929 20:17:02.684134 2101 ts_manager.cc:194] Registered new tserver with Master: eaa3fccfe54b400e80a3f4c5c33a1baf (127.27.185.65:42717)
I20260929 20:17:02.685696 2101 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.185.65@KRBTEST.COM'} at 127.27.185.65:52685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.66.keytab.
Entry for principal oryx/127.27.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.66.keytab.
I20260929 20:17:02.707567 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:0
--local_ip_for_outbound_sockets=127.27.185.66
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:36737
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.66.keytab
--principal=oryx/127.27.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:40591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:17:03.035902 2296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:03.036286 2296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:03.036477 2296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:03.043651 2296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:03.043820 2296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:03.044016 2296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:17:03.055411 2296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:40591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.66.keytab
--principal=oryx/127.27.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:36737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:03.056915 2296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:03.058420 2296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:03.073002 2302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:03.073087 2301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:03.073069 2304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:03.074092 2296 server_base.cc:1034] running on GCE node
I20260929 20:17:03.074725 2296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:03.075798 2296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:03.077039 2296 hybrid_clock.cc:648] HybridClock initialized: now 1790713023076978 us; error 51 us; skew 500 ppm
I20260929 20:17:03.077409 2296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:17:03 dist-test-slave-4cx3 krb5kdc[2056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713023, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:17:03.080749 2296 init.cc:377] Logged in from keytab as oryx/127.27.185.66@KRBTEST.COM (short username oryx)
I20260929 20:17:03.082196 2296 webserver.cc:460] Webserver started at http://127.27.185.66:35653/ using document root <none> and password file <none>
I20260929 20:17:03.082850 2296 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:03.082952 2296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:03.083199 2296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:17:03.086076 2296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/data/instance:
uuid: "100246ec987b437eb04e30cf2815bced"
format_stamp: "Formatted at 2026-09-29 20:17:03 on dist-test-slave-4cx3"
server_key: "0adb6320d9ce3a8a945349515a13ca79"
server_key_iv: "fc12aa43537c47c1596ef9a7e0b4a646"
server_key_version: "encryptionkey@0"
I20260929 20:17:03.086793 2296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance:
uuid: "100246ec987b437eb04e30cf2815bced"
format_stamp: "Formatted at 2026-09-29 20:17:03 on dist-test-slave-4cx3"
server_key: "0adb6320d9ce3a8a945349515a13ca79"
server_key_iv: "fc12aa43537c47c1596ef9a7e0b4a646"
server_key_version: "encryptionkey@0"
I20260929 20:17:03.092025 2296 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 20:17:03.095551 2311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:03.096845 2296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:17:03.097014 2296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "100246ec987b437eb04e30cf2815bced"
format_stamp: "Formatted at 2026-09-29 20:17:03 on dist-test-slave-4cx3"
server_key: "0adb6320d9ce3a8a945349515a13ca79"
server_key_iv: "fc12aa43537c47c1596ef9a7e0b4a646"
server_key_version: "encryptionkey@0"
I20260929 20:17:03.097250 2296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:03.132812 2296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:03.136847 2296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:03.137151 2296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:03.138176 2296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:17:03.139678 2296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:17:03.139748 2296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:03.139905 2296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:17:03.139963 2296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:03.173483 2296 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:33241
I20260929 20:17:03.173529 2423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:33241 every 8 connection(s)
I20260929 20:17:03.174867 2296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:17:03.178939 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 2296
I20260929 20:17:03.179159 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance
I20260929 20:17:03.179699 28389 external_mini_cluster.cc:1428] Setting key 20f1490af3e410a0be79637b7039e053
Sep 29 20:17:03 dist-test-slave-4cx3 krb5kdc[2056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713023, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM
I20260929 20:17:03.197320 2426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:03.177553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:34999 (local address 127.27.185.126:36737)
0929 20:17:03.177979 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:03.177984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:03.179236 (+ 1252us) server_negotiation.cc:407] Connection header received
0929 20:17:03.181238 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:03.181244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:03.181356 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:03.181479 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:03.183026 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:03.183856 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:03.185447 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:03.185849 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:03.185991 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:03.190015 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:03.190025 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:03.190032 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:03.190080 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:03.193878 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:03.194729 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:03.194737 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:03.194743 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:03.194902 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:03.195322 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:03.195330 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:03.195336 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:03.195769 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:03.195929 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:03.196601 (+ 672us) server_negotiation.cc:299] Negotiation successful
0929 20:17:03.196879 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":200,"threads_started":1}
I20260929 20:17:03.198096 2427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:03.178199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.27.185.66:34999)
0929 20:17:03.179052 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:03.179104 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:03.180957 (+ 1853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:03.181741 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:03.181750 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:03.182270 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:03.182844 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:03.182868 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:03.184019 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:03.184024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:03.185278 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:03.185290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:03.186086 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:03.186094 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:03.186345 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:03.187031 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:03.187070 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:03.189774 (+ 2704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:03.194053 (+ 4279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:03.194067 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:03.194081 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:03.194597 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:03.195019 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:03.195022 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:03.195024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:03.195208 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:03.195890 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:03.195896 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:03.196235 (+ 339us) client_negotiation.cc:769] Sending connection context
0929 20:17:03.196471 (+ 236us) client_negotiation.cc:240] Negotiation successful
0929 20:17:03.196965 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":349,"threads_started":1}
I20260929 20:17:03.199651 2424 heartbeater.cc:347] Connected to a master server at 127.27.185.126:36737
I20260929 20:17:03.199954 2424 heartbeater.cc:464] Registering TS with master...
I20260929 20:17:03.200780 2424 heartbeater.cc:511] Master 127.27.185.126:36737 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.27.185.67@KRBTEST.COM; defaulting to no policy
I20260929 20:17:03.202359 2101 ts_manager.cc:194] Registered new tserver with Master: 100246ec987b437eb04e30cf2815bced (127.27.185.66:33241)
I20260929 20:17:03.203292 2101 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.185.66@KRBTEST.COM'} at 127.27.185.66:34999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.67.keytab.
Entry for principal oryx/127.27.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.67.keytab.
I20260929 20:17:03.228806 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:0
--local_ip_for_outbound_sockets=127.27.185.67
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:36737
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.67.keytab
--principal=oryx/127.27.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:40591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:17:03.551649 2430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:03.551971 2430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:03.552158 2430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:03.559422 2430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:03.559583 2430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:03.559769 2430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:17:03.569690 2430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:40591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.67.keytab
--principal=oryx/127.27.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:36737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:03.571066 2430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:03.572438 2430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:03.586618 2438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:03.586669 2436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:03.586783 2435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:03.588021 2430 server_base.cc:1034] running on GCE node
I20260929 20:17:03.588752 2430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:03.589752 2430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:03.591001 2430 hybrid_clock.cc:648] HybridClock initialized: now 1790713023590944 us; error 48 us; skew 500 ppm
I20260929 20:17:03.591334 2430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:17:03 dist-test-slave-4cx3 krb5kdc[2056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713023, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:17:03.594605 2430 init.cc:377] Logged in from keytab as oryx/127.27.185.67@KRBTEST.COM (short username oryx)
I20260929 20:17:03.596182 2430 webserver.cc:460] Webserver started at http://127.27.185.67:42741/ using document root <none> and password file <none>
I20260929 20:17:03.596978 2430 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:03.597129 2430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:03.597442 2430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:17:03.600878 2430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/data/instance:
uuid: "67b9089a4cb642d682558e47961dcc63"
format_stamp: "Formatted at 2026-09-29 20:17:03 on dist-test-slave-4cx3"
server_key: "9238fd013f7fdd5ee461b2c913b29181"
server_key_iv: "fb5aa22b9b969e442e0a7f1245aa5fb6"
server_key_version: "encryptionkey@0"
I20260929 20:17:03.601740 2430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance:
uuid: "67b9089a4cb642d682558e47961dcc63"
format_stamp: "Formatted at 2026-09-29 20:17:03 on dist-test-slave-4cx3"
server_key: "9238fd013f7fdd5ee461b2c913b29181"
server_key_iv: "fb5aa22b9b969e442e0a7f1245aa5fb6"
server_key_version: "encryptionkey@0"
I20260929 20:17:03.607718 2430 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 20:17:03.611636 2445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:03.613315 2430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:17:03.613547 2430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "67b9089a4cb642d682558e47961dcc63"
format_stamp: "Formatted at 2026-09-29 20:17:03 on dist-test-slave-4cx3"
server_key: "9238fd013f7fdd5ee461b2c913b29181"
server_key_iv: "fb5aa22b9b969e442e0a7f1245aa5fb6"
server_key_version: "encryptionkey@0"
I20260929 20:17:03.613766 2430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:03.652833 2430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:03.657356 2430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:03.657677 2430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:03.658762 2430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:17:03.660215 2430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:17:03.660332 2430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:03.660486 2430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:17:03.660559 2430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:03.689050 2290 heartbeater.cc:503] Master 127.27.185.126:36737 was elected leader, sending a full tablet report...
I20260929 20:17:03.695981 2430 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:45841
I20260929 20:17:03.696070 2557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:45841 every 8 connection(s)
I20260929 20:17:03.697299 2430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:17:03.699110 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 2430
I20260929 20:17:03.699481 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance
I20260929 20:17:03.700074 28389 external_mini_cluster.cc:1428] Setting key b812d72b1555f774ce4b98e33998bbab
Sep 29 20:17:03 dist-test-slave-4cx3 krb5kdc[2056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713023, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM
I20260929 20:17:03.718312 2560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:03.700078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:41687 (local address 127.27.185.126:36737)
0929 20:17:03.700608 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:03.700617 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:03.702056 (+ 1439us) server_negotiation.cc:407] Connection header received
0929 20:17:03.704202 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:03.704206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:03.704288 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:03.704379 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:03.705865 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:03.706649 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:03.708253 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:03.708646 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:03.708749 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:03.712166 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:03.712171 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:03.712174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:03.712206 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:03.715235 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:03.716209 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:03.716214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:03.716216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:03.716336 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:03.716707 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:03.716710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:03.716712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:03.716981 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:03.717097 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:03.717854 (+ 757us) server_negotiation.cc:299] Negotiation successful
0929 20:17:03.718033 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":247,"threads_started":1}
I20260929 20:17:03.719695 2561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:03.700948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.27.185.67:41687)
0929 20:17:03.701870 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:03.701948 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:03.703964 (+ 2016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:03.704600 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:03.704610 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:03.705127 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:03.705707 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:03.705724 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:03.706779 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:03.706783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:03.708106 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:03.708118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:03.708829 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:03.708834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:03.709042 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:03.709679 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:03.709701 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:03.711963 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:03.715385 (+ 3422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:03.715419 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:03.715438 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:03.716091 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:03.716424 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:03.716427 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:03.716429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:03.716606 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:03.717083 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:03.717089 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:03.717492 (+ 403us) client_negotiation.cc:769] Sending connection context
0929 20:17:03.717727 (+ 235us) client_negotiation.cc:240] Negotiation successful
0929 20:17:03.718319 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":346,"threads_started":1}
I20260929 20:17:03.721127 2558 heartbeater.cc:347] Connected to a master server at 127.27.185.126:36737
I20260929 20:17:03.721469 2558 heartbeater.cc:464] Registering TS with master...
I20260929 20:17:03.722244 2558 heartbeater.cc:511] Master 127.27.185.126:36737 requested a full tablet report, sending...
I20260929 20:17:03.723723 2101 ts_manager.cc:194] Registered new tserver with Master: 67b9089a4cb642d682558e47961dcc63 (127.27.185.67:45841)
I20260929 20:17:03.724500 2101 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.185.67@KRBTEST.COM'} at 127.27.185.67:41687
I20260929 20:17:03.728359 28389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.185.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.185.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.185.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.125.keytab.
Entry for principal oryx/127.27.185.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.125.keytab.
I20260929 20:17:03.779335 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.125:46841
--webserver_interface=127.27.185.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.125.keytab
--principal=oryx/127.27.185.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:40591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:36737,127.27.185.125:46841
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:17:04.109754 2564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:04.110098 2564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:04.110251 2564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:04.117888 2564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:17:04.118031 2564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:04.118110 2564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:04.118198 2564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:17:04.118266 2564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:17:04.128489 2564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:40591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:36737,127.27.185.125:46841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.125.keytab
--principal=oryx/127.27.185.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.125:46841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.185.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
W20260929 20:17:04.128894 2564 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.27.185.126:36737,127.27.185.125:46841'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 20:17:04.130024 2564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:04.131347 2564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:04.143195 2569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:04.143110 2572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:04.143170 2570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:04.143944 2564 server_base.cc:1034] running on GCE node
I20260929 20:17:04.144701 2564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:04.146049 2564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:04.147262 2564 hybrid_clock.cc:648] HybridClock initialized: now 1790713024147211 us; error 24 us; skew 500 ppm
I20260929 20:17:04.147665 2564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:17:04 dist-test-slave-4cx3 krb5kdc[2056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713024, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:17:04.150885 2564 init.cc:377] Logged in from keytab as oryx/127.27.185.125@KRBTEST.COM (short username oryx)
I20260929 20:17:04.152612 2564 webserver.cc:460] Webserver started at http://127.27.185.125:40947/ using document root <none> and password file <none>
I20260929 20:17:04.153378 2564 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:04.153489 2564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:04.153748 2564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:17:04.156679 2564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-1/data/instance:
uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49"
format_stamp: "Formatted at 2026-09-29 20:17:04 on dist-test-slave-4cx3"
server_key: "04b07d1e3bf2f9098fed1a1d6e871df4"
server_key_iv: "e4ead8efb33238233e23dc78c3283c72"
server_key_version: "encryptionkey@0"
I20260929 20:17:04.157378 2564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-1/wal/instance:
uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49"
format_stamp: "Formatted at 2026-09-29 20:17:04 on dist-test-slave-4cx3"
server_key: "04b07d1e3bf2f9098fed1a1d6e871df4"
server_key_iv: "e4ead8efb33238233e23dc78c3283c72"
server_key_version: "encryptionkey@0"
I20260929 20:17:04.162403 2564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 20:17:04.165923 2579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:04.167246 2564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 20:17:04.167512 2564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-1/wal
uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49"
format_stamp: "Formatted at 2026-09-29 20:17:04 on dist-test-slave-4cx3"
server_key: "04b07d1e3bf2f9098fed1a1d6e871df4"
server_key_iv: "e4ead8efb33238233e23dc78c3283c72"
server_key_version: "encryptionkey@0"
I20260929 20:17:04.167724 2564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:04.190063 2564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:04.194814 2564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:04.195087 2564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:04.205830 2424 heartbeater.cc:503] Master 127.27.185.126:36737 was elected leader, sending a full tablet report...
I20260929 20:17:04.214128 2564 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.125:46841
I20260929 20:17:04.214149 2630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.125:46841 every 8 connection(s)
I20260929 20:17:04.215536 2564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-1/data/info.pb
I20260929 20:17:04.219529 2631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:04.219857 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 2564
I20260929 20:17:04.229293 2631 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:17:04 dist-test-slave-4cx3 krb5kdc[2056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.185.125@KRBTEST.COM
Sep 29 20:17:04 dist-test-slave-4cx3 krb5kdc[2056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713024, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.125@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM
I20260929 20:17:04.243062 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:04.222197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34212 (local address 127.27.185.125:46841)
0929 20:17:04.223029 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:04.223042 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:04.223084 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:17:04.224698 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:04.224720 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:04.225113 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:04.225433 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:04.227228 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.228582 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.230354 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.231016 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.231136 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:04.236242 (+ 5106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:04.236253 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:04.236262 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:04.236304 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:04.239117 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:04.239885 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:04.239896 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:04.239912 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:04.240079 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:04.240451 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:04.240457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:04.240463 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:04.241030 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:04.241354 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:04.241769 (+ 415us) server_negotiation.cc:299] Negotiation successful
0929 20:17:04.242150 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":398,"threads_started":1}
I20260929 20:17:04.246946 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:04.232257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34488 (local address 127.27.185.126:36737)
0929 20:17:04.233592 (+ 1335us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:04.233601 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:04.233622 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:17:04.233688 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:04.233695 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:04.233790 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:04.233913 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:04.235037 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.235857 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.237461 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.238025 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.238174 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:04.241309 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:04.241316 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:04.241320 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:04.241373 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:04.244189 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:04.244868 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:04.244873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:04.244876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:04.245032 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:04.245377 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:04.245381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:04.245384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:04.245773 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:04.245928 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:04.246385 (+ 457us) server_negotiation.cc:299] Negotiation successful
0929 20:17:04.246599 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1195,"thread_start_us":1082,"threads_started":1}
I20260929 20:17:04.246986 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:04.232186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.0.0.1:34488)
0929 20:17:04.232896 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:04.232945 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:04.233124 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:04.233980 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:04.233987 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:04.234540 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:04.234874 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:04.234886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.235996 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.236003 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:04.237273 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:04.237288 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.238212 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.238219 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:04.238407 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:04.239009 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:04.239032 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:04.241124 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:04.244307 (+ 3183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:04.244314 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:04.244319 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:04.244749 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:04.245143 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:04.245145 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:04.245148 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:04.245271 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:04.245916 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:04.245924 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:04.246088 (+ 164us) client_negotiation.cc:769] Sending connection context
0929 20:17:04.246277 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 20:17:04.246591 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":290,"threads_started":1}
I20260929 20:17:04.248564 2631 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:17:04 dist-test-slave-4cx3 krb5kdc[2056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713024, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.125@KRBTEST.COM for oryx/127.27.185.125@KRBTEST.COM
I20260929 20:17:04.259681 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:04.249171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:46841 (local address 127.0.0.1:34218)
0929 20:17:04.249331 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:04.249351 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:04.249427 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:04.249688 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:04.249691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:04.249982 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:04.250103 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:04.250110 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.250994 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.250997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:04.251905 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:04.251916 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.252544 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.252548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:04.252688 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:04.253229 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:04.253246 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:04.255155 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:04.257706 (+ 2551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:04.257709 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:04.257712 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:04.258053 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:04.258342 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:04.258345 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:04.258348 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:04.258480 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:04.258932 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:04.258935 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:04.259050 (+ 115us) client_negotiation.cc:769] Sending connection context
0929 20:17:04.259155 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 20:17:04.259343 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 20:17:04.259817 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:04.249346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34218 (local address 127.27.185.125:46841)
0929 20:17:04.249482 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:04.249487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:04.249500 (+ 13us) server_negotiation.cc:407] Connection header received
0929 20:17:04.249534 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:04.249537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:04.249584 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:04.249656 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:04.250213 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.250867 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.252025 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.252409 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.252502 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:04.255307 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:04.255311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:04.255315 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:04.255341 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:04.257587 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:04.258151 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:04.258155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:04.258157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:04.258254 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:04.258565 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:04.258568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:04.258570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:04.258824 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:04.258929 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:04.259232 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 20:17:04.259530 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":48}
I20260929 20:17:04.264353 2631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49: Bootstrap starting.
I20260929 20:17:04.267282 2631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:04.268498 2631 log.cc:824] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49: Log is configured to *not* fsync() on all Append() calls
I20260929 20:17:04.271082 2631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49: No bootstrap required, opened a new log
I20260929 20:17:04.278218 2631 raft_consensus.cc:348] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } }
I20260929 20:17:04.278544 2631 raft_consensus.cc:374] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:04.278638 2631 raft_consensus.cc:729] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb7a8c34c984b92ab34d3a2e1d14d49, State: Initialized, Role: FOLLOWER
I20260929 20:17:04.279250 2631 consensus_queue.cc:260] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } }
I20260929 20:17:04.280121 2638 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } } }
I20260929 20:17:04.280409 2638 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:17:04.281095 2631 sys_catalog.cc:564] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 20:17:04.289166 2649 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:17:04.289319 2649 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:17:04.292116 2564 master_runner.cc:428] Detected that this master ceb7a8c34c984b92ab34d3a2e1d14d49 is joining an existing cluster
I20260929 20:17:04.292223 2564 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.185.125:46841
I20260929 20:17:04.315354 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:04.296771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34234 (local address 127.27.185.125:46841)
0929 20:17:04.297012 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:04.297020 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:04.297632 (+ 612us) server_negotiation.cc:407] Connection header received
0929 20:17:04.297880 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:04.297891 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:04.297991 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:04.298079 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:04.299906 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.301381 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.304091 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.305058 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.305179 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:04.307804 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:04.307810 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:04.307815 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:04.307847 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:04.311638 (+ 3791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:04.312788 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:04.312793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:04.312796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:04.312948 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:04.313434 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:04.313439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:04.313443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:04.313807 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:04.313948 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:04.314443 (+ 495us) server_negotiation.cc:299] Negotiation successful
0929 20:17:04.314920 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:17:04.317880 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:04.296381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:46841 (local address 127.0.0.1:34234)
0929 20:17:04.297511 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:04.297537 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:04.297734 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:04.298324 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:04.298332 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:04.299195 (+ 863us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:04.299720 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:04.299737 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.301730 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.301737 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:04.303901 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:04.303920 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.305448 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.305453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:04.305646 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:04.306393 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:04.306416 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:04.307634 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:04.311784 (+ 4150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:04.311789 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:04.311793 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:04.312658 (+ 865us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:04.313094 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:04.313098 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:04.313102 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:04.313309 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:04.314032 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:04.314038 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:04.314191 (+ 153us) client_negotiation.cc:769] Sending connection context
0929 20:17:04.315619 (+ 1428us) client_negotiation.cc:240] Negotiation successful
0929 20:17:04.317374 (+ 1755us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"thread_start_us":822,"threads_started":1}
I20260929 20:17:04.320508 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:04.296190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34496 (local address 127.27.185.126:36737)
0929 20:17:04.296382 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:04.296388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:04.305029 (+ 8641us) server_negotiation.cc:407] Connection header received
0929 20:17:04.305439 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:04.305445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:04.305551 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:04.305625 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:04.307253 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.308361 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.310609 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.311265 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.311369 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:04.313904 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:04.313919 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:04.313924 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:04.313964 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:04.316945 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:04.317823 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:04.317829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:04.317833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:04.318433 (+ 600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:04.318957 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:04.318962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:04.318965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:04.319304 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:04.319471 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:04.320082 (+ 611us) server_negotiation.cc:299] Negotiation successful
0929 20:17:04.320270 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:17:04.321357 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:04.297730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.0.0.1:34496)
0929 20:17:04.304913 (+ 7183us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:04.304935 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:04.305271 (+ 336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:04.305842 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:04.305849 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:04.306654 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:04.307069 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:04.307083 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.308581 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.308592 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:04.310426 (+ 1834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:04.310440 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.311711 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.311719 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:04.311907 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:04.312561 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:04.312587 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:04.313684 (+ 1097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:04.317095 (+ 3411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:04.317100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:04.317104 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:04.317679 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:04.318645 (+ 966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:04.318649 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:04.318653 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:04.318820 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:04.319569 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:04.319577 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:04.319782 (+ 205us) client_negotiation.cc:769] Sending connection context
0929 20:17:04.320769 (+ 987us) client_negotiation.cc:240] Negotiation successful
0929 20:17:04.321075 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7021,"thread_start_us":6645,"threads_started":1}
WARNING: no policy specified for oryx/127.27.185.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.185.124@KRBTEST.COM" created.
Sep 29 20:17:04 dist-test-slave-4cx3 krb5kdc[2056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713021, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.126@KRBTEST.COM for oryx/127.27.185.125@KRBTEST.COM
I20260929 20:17:04.344107 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:04.327827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:46841 (local address 127.0.0.1:34250)
0929 20:17:04.328520 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:04.328573 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:04.328816 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:04.329353 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:04.329362 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:04.330227 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:04.330518 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:04.330534 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.332073 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.332079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:04.333682 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:04.333697 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.334499 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.334506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:04.334702 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:04.335342 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:04.335360 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:04.338087 (+ 2727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:04.341304 (+ 3217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:04.341316 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:04.341323 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:04.341740 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:04.342135 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:04.342140 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:04.342144 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:04.342289 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:04.342835 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:04.342843 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:04.343020 (+ 177us) client_negotiation.cc:769] Sending connection context
0929 20:17:04.343222 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 20:17:04.343653 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":293,"threads_started":1}
I20260929 20:17:04.344153 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:04.327728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34250 (local address 127.27.185.125:46841)
0929 20:17:04.327900 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:04.327906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:04.328659 (+ 753us) server_negotiation.cc:407] Connection header received
0929 20:17:04.329052 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:04.329060 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:04.329167 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:04.329273 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:04.330710 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.331929 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.333864 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.334325 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.334467 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:04.338276 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:04.338281 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:04.338284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:04.338316 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:04.341159 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:04.341870 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:04.341877 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:04.341886 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:04.342018 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:04.342399 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:04.342406 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:04.342409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:04.342689 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:04.342855 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:04.343597 (+ 742us) server_negotiation.cc:299] Negotiation successful
0929 20:17:04.343804 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:17:04.346388 2101 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.185.125:46841: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49"
member_type: NON_VOTER
last_known_addr {
host: "127.27.185.125"
port: 46841
}
attrs {
promote: true
}
}
dest_uuid: "3a60def670884f8886ada0b1616e4159"
cas_config_opid_index: -1
I20260929 20:17:04.347368 2101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: NON_VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: true } }
I20260929 20:17:04.350931 2662 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ceb7a8c34c984b92ab34d3a2e1d14d49 (127.27.185.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: NON_VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: true } } }
I20260929 20:17:04.352403 2606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:17:04.355782 2662 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.185.125:46841
I20260929 20:17:04.356607 2564 master.cc:537] Master@127.27.185.125:46841 shutting down...
I20260929 20:17:04.356416 2661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3a60def670884f8886ada0b1616e4159" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: NON_VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: true } } }
I20260929 20:17:04.356992 2661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.185.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.124.keytab.
Entry for principal oryx/127.27.185.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.124.keytab.
I20260929 20:17:04.359100 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.124:42155
--webserver_interface=127.27.185.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.124.keytab
--principal=oryx/127.27.185.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:40591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:36737,127.27.185.125:46841,127.27.185.124:42155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:17:04.365032 2606 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [term 1 FOLLOWER]: Refusing update from remote peer 3a60def670884f8886ada0b1616e4159: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 20:17:04.369050 2662 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: NON_VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:17:04.371698 2564 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:17:04.372150 2564 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:17:04.372368 2564 tablet_replica.cc:330] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49: stopping tablet replica
W20260929 20:17:04.372515 2085 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260929 20:17:04.379794 2564 master.cc:552] Master@127.27.185.125:46841 shutdown complete.
I20260929 20:17:04.379889 2564 master_runner.cc:305] Clearing existing system tablet
I20260929 20:17:04.380427 2564 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:04.380578 2564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:04.384013 2564 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 20:17:04.384310 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:04.383782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:46841 (local address 127.0.0.1:34256)
0929 20:17:04.384015 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:04.384161 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.125:46841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260929 20:17:04.384550 2085 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 -> Peer ceb7a8c34c984b92ab34d3a2e1d14d49 (127.27.185.125:46841): Couldn't send request to peer ceb7a8c34c984b92ab34d3a2e1d14d49. Status: Network error: Client connection negotiation failed: client connection to 127.27.185.125:46841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 20:17:04.385728 2667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:04.386076 2564 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:04.386561 2564 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 20:17:04.386695 2564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-1/wal
uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49"
format_stamp: "Formatted at 2026-09-29 20:17:04 on dist-test-slave-4cx3"
server_key: "04b07d1e3bf2f9098fed1a1d6e871df4"
server_key_iv: "e4ead8efb33238233e23dc78c3283c72"
server_key_version: "encryptionkey@0"
I20260929 20:17:04.391945 2564 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:17:04.398937 2564 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 20:17:04.399200 2564 log.cc:1196] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 20:17:04.399708 2564 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49: Deleting consensus metadata
I20260929 20:17:04.400030 2564 master_runner.cc:315] Copying system tablet from 127.27.185.126:36737
I20260929 20:17:04.402853 2564 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49: tablet copy: Beginning tablet copy session from remote peer at address 127.27.185.126:36737
I20260929 20:17:04.419437 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:04.403770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34506 (local address 127.27.185.126:36737)
0929 20:17:04.403968 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:04.403976 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:04.404462 (+ 486us) server_negotiation.cc:407] Connection header received
0929 20:17:04.404744 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:04.404753 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:04.404848 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:04.404921 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:04.406137 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.407304 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.409230 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.409948 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.410093 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:04.412290 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:04.412319 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:04.412324 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:04.412369 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:04.416176 (+ 3807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:04.416967 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:04.416972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:04.416976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:04.417137 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:04.417566 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:04.417571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:04.417574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:04.418024 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:04.418208 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:04.418694 (+ 486us) server_negotiation.cc:299] Negotiation successful
0929 20:17:04.419015 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:17:04.419835 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:04.403729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.0.0.1:34506)
0929 20:17:04.404344 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:04.404364 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:04.404581 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:04.405162 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:04.405167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:04.405665 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:04.405947 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:04.405969 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.407561 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.407566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:04.409068 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:04.409082 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.410130 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.410136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:04.410297 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:04.411005 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:04.411028 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:04.412078 (+ 1050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:04.416323 (+ 4245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:04.416329 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:04.416333 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:04.416831 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:04.417259 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:04.417264 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:04.417268 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:04.417449 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:04.418166 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:04.418172 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:04.418384 (+ 212us) client_negotiation.cc:769] Sending connection context
0929 20:17:04.418558 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 20:17:04.419014 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":354,"threads_started":1}
I20260929 20:17:04.420790 2121 tablet_copy_service.cc:140] P 3a60def670884f8886ada0b1616e4159: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ceb7a8c34c984b92ab34d3a2e1d14d49 ({username='oryx', principal='oryx/127.27.185.125@KRBTEST.COM'} at 127.0.0.1:34506)
I20260929 20:17:04.421124 2121 tablet_copy_service.cc:161] P 3a60def670884f8886ada0b1616e4159: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ceb7a8c34c984b92ab34d3a2e1d14d49 at {username='oryx', principal='oryx/127.27.185.125@KRBTEST.COM'} at 127.0.0.1:34506: session id = ceb7a8c34c984b92ab34d3a2e1d14d49-00000000000000000000000000000000
I20260929 20:17:04.426160 2121 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 20:17:04.429003 2564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 20:17:04.435943 2564 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49: tablet copy: Starting download of 0 data blocks...
I20260929 20:17:04.436273 2564 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49: tablet copy: Starting download of 1 WAL segments...
I20260929 20:17:04.439193 2564 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 20:17:04.443608 2121 tablet_copy_service.cc:342] P 3a60def670884f8886ada0b1616e4159: Request end of tablet copy session ceb7a8c34c984b92ab34d3a2e1d14d49-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.185.125@KRBTEST.COM'} at 127.0.0.1:34506
I20260929 20:17:04.443835 2121 tablet_copy_service.cc:433] P 3a60def670884f8886ada0b1616e4159: ending tablet copy session ceb7a8c34c984b92ab34d3a2e1d14d49-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ceb7a8c34c984b92ab34d3a2e1d14d49
I20260929 20:17:04.456503 2564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:04.459306 2674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:04.465054 2675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:04.469563 2677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:04.469933 2564 server_base.cc:1034] running on GCE node
I20260929 20:17:04.470270 2564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:04.470924 2564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:04.472208 2564 hybrid_clock.cc:648] HybridClock initialized: now 1790713024472140 us; error 44 us; skew 500 ppm
I20260929 20:17:04.472440 2564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:17:04 dist-test-slave-4cx3 krb5kdc[2056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713024, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:17:04.474715 2564 init.cc:377] Logged in from keytab as oryx/127.27.185.125@KRBTEST.COM (short username oryx)
I20260929 20:17:04.477365 2564 webserver.cc:460] Webserver started at http://127.27.185.125:34833/ using document root <none> and password file <none>
I20260929 20:17:04.477800 2564 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:04.477890 2564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:04.481046 2564 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260929 20:17:04.483018 2684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:04.483515 2564 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 20:17:04.483702 2564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-1/wal
uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49"
format_stamp: "Formatted at 2026-09-29 20:17:04 on dist-test-slave-4cx3"
server_key: "04b07d1e3bf2f9098fed1a1d6e871df4"
server_key_iv: "e4ead8efb33238233e23dc78c3283c72"
server_key_version: "encryptionkey@0"
I20260929 20:17:04.483946 2564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:04.532470 2564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:04.537321 2564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:04.565279 2564 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.125:46841
I20260929 20:17:04.565297 2735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.125:46841 every 8 connection(s)
I20260929 20:17:04.566618 2564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-1/data/info.pb
I20260929 20:17:04.572710 2736 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:17:04.577198 2736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49: Bootstrap starting.
I20260929 20:17:04.596338 2736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:17:04.596845 2736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49: Bootstrap complete.
I20260929 20:17:04.597913 2736 raft_consensus.cc:348] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: NON_VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: true } }
I20260929 20:17:04.598181 2736 raft_consensus.cc:729] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ceb7a8c34c984b92ab34d3a2e1d14d49, State: Initialized, Role: LEARNER
I20260929 20:17:04.598424 2736 consensus_queue.cc:260] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: NON_VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: true } }
I20260929 20:17:04.599254 2739 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: NON_VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: true } } }
I20260929 20:17:04.599529 2736 sys_catalog.cc:564] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:17:04.599709 2739 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [sys.catalog]: This master's current role is: LEARNER
I20260929 20:17:04.611075 2750 catalog_manager.cc:1259] Loaded cluster ID: 09b63d71bcc342fcaf7a45474598022f
I20260929 20:17:04.611193 2750 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49: loading cluster ID for follower catalog manager: success
I20260929 20:17:04.614800 2750 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49: acquiring CA information for follower catalog manager: success
I20260929 20:17:04.617733 2750 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 20:17:04.727180 2558 heartbeater.cc:503] Master 127.27.185.126:36737 was elected leader, sending a full tablet report...
W20260929 20:17:04.754468 2664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:04.754791 2664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:04.754948 2664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:04.762413 2664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:17:04.762531 2664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:04.762616 2664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:04.762722 2664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:17:04.762782 2664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:17:04.773113 2664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:40591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:36737,127.27.185.125:46841,127.27.185.124:42155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.124.keytab
--principal=oryx/127.27.185.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.124:42155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.185.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:04.774451 2664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:04.775832 2664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:04.787344 2755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:04.787567 2756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:04.787547 2758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:04.788262 2664 server_base.cc:1034] running on GCE node
I20260929 20:17:04.789011 2664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:04.790452 2664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:04.791642 2664 hybrid_clock.cc:648] HybridClock initialized: now 1790713024791586 us; error 48 us; skew 500 ppm
I20260929 20:17:04.791966 2664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:17:04 dist-test-slave-4cx3 krb5kdc[2056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713024, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:17:04.795080 2664 init.cc:377] Logged in from keytab as oryx/127.27.185.124@KRBTEST.COM (short username oryx)
I20260929 20:17:04.796583 2664 webserver.cc:460] Webserver started at http://127.27.185.124:35011/ using document root <none> and password file <none>
I20260929 20:17:04.797240 2664 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:04.797380 2664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:04.797674 2664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:17:04.800895 2664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-2/data/instance:
uuid: "927955a2d44a4c7198c39f371d6671d3"
format_stamp: "Formatted at 2026-09-29 20:17:04 on dist-test-slave-4cx3"
server_key: "f38f0052626cfb3e01bd9fb351609900"
server_key_iv: "a8b6bbcfb52b1eead0e74a994be3e274"
server_key_version: "encryptionkey@0"
I20260929 20:17:04.801702 2664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-2/wal/instance:
uuid: "927955a2d44a4c7198c39f371d6671d3"
format_stamp: "Formatted at 2026-09-29 20:17:04 on dist-test-slave-4cx3"
server_key: "f38f0052626cfb3e01bd9fb351609900"
server_key_iv: "a8b6bbcfb52b1eead0e74a994be3e274"
server_key_version: "encryptionkey@0"
I20260929 20:17:04.806862 2664 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:17:04.810349 2765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:04.811813 2664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:17:04.812009 2664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-2/wal
uuid: "927955a2d44a4c7198c39f371d6671d3"
format_stamp: "Formatted at 2026-09-29 20:17:04 on dist-test-slave-4cx3"
server_key: "f38f0052626cfb3e01bd9fb351609900"
server_key_iv: "a8b6bbcfb52b1eead0e74a994be3e274"
server_key_version: "encryptionkey@0"
I20260929 20:17:04.812312 2664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:04.836805 2664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:04.840883 2664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:04.841192 2664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:04.854526 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:04.847206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:46841 (local address 127.0.0.1:34266)
0929 20:17:04.847574 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:04.847601 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:04.847743 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:04.848744 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:04.848751 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:04.848782 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:17:04.848978 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:04.848988 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.850525 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.850531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:04.852303 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:04.852316 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.853643 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.853651 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:04.853864 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:04.853939 (+ 75us) client_negotiation.cc:769] Sending connection context
0929 20:17:04.854050 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 20:17:04.854201 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260929 20:17:04.854528 2778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:04.847468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34266 (local address 127.27.185.125:46841)
0929 20:17:04.848321 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:04.848328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:04.848355 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:17:04.848438 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:04.848444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:04.848552 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:04.848696 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:17:04.849147 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.850303 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.852449 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.853459 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.853666 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:04.853864 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:04.854086 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 20:17:04.854218 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":559,"threads_started":1}
I20260929 20:17:04.856015 2711 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260929 20:17:04.856690 2739 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a60def670884f8886ada0b1616e4159. Latest consensus state: current_term: 1 leader_uuid: "3a60def670884f8886ada0b1616e4159" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: NON_VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: true } } }
I20260929 20:17:04.856983 2739 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [sys.catalog]: This master's current role is: LEARNER
I20260929 20:17:04.861935 2817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.124:42155 every 8 connection(s)
I20260929 20:17:04.861941 2664 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.124:42155
I20260929 20:17:04.863344 2664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-2/data/info.pb
I20260929 20:17:04.867043 2818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:04.871428 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 2664
I20260929 20:17:04.875005 2818 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:17:04 dist-test-slave-4cx3 krb5kdc[2056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.185.124@KRBTEST.COM
Sep 29 20:17:04 dist-test-slave-4cx3 krb5kdc[2056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713024, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.124@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM
I20260929 20:17:04.894219 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:04.878840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34508 (local address 127.27.185.126:36737)
0929 20:17:04.879049 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:04.879057 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:04.879819 (+ 762us) server_negotiation.cc:407] Connection header received
0929 20:17:04.879996 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:04.880001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:04.880081 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:04.880186 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:04.881246 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.882257 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.884265 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.884888 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.885047 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:04.888479 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:04.888484 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:04.888487 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:04.888520 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:04.891242 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:04.892028 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:04.892032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:04.892034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:04.892157 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:04.892557 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:04.892562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:04.892564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:04.892814 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:04.892983 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:04.893608 (+ 625us) server_negotiation.cc:299] Negotiation successful
0929 20:17:04.893822 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:17:04.895141 2821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:04.873731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54946 (local address 127.27.185.124:42155)
0929 20:17:04.874482 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:04.874493 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:04.874532 (+ 39us) server_negotiation.cc:407] Connection header received
0929 20:17:04.876241 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:04.876265 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:04.876731 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:04.877191 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:04.878806 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.880595 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.882170 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.883191 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.883374 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:04.888232 (+ 4858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:04.888240 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:04.888252 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:04.888284 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:04.890890 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:04.891628 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:04.891644 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:04.891646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:04.891832 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:04.892207 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:04.892213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:04.892216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:04.892704 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:04.893025 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:04.893449 (+ 424us) server_negotiation.cc:299] Negotiation successful
0929 20:17:04.893886 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":556,"thread_start_us":355,"threads_started":1}
I20260929 20:17:04.895187 2822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:04.878828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.0.0.1:34508)
0929 20:17:04.879678 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:04.879718 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:04.879866 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:04.880245 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:04.880250 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:04.880850 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:04.881094 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:04.881102 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.882383 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.882388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:04.884100 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:04.884115 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.885134 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.885141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:04.885261 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:04.885836 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:04.885862 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:04.888277 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:04.891429 (+ 3152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:04.891440 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:04.891451 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:04.891883 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:04.892272 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:04.892277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:04.892280 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:04.892429 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:04.892954 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:04.893015 (+ 61us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:04.893243 (+ 228us) client_negotiation.cc:769] Sending connection context
0929 20:17:04.893494 (+ 251us) client_negotiation.cc:240] Negotiation successful
0929 20:17:04.894069 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":132,"thread_start_us":418,"threads_started":1}
I20260929 20:17:04.896744 2818 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:17:04 dist-test-slave-4cx3 krb5kdc[2056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713024, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.124@KRBTEST.COM for oryx/127.27.185.125@KRBTEST.COM
I20260929 20:17:04.909068 2822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:04.897488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:46841 (local address 127.0.0.1:34280)
0929 20:17:04.897897 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:04.897934 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:04.898044 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:04.898283 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:04.898286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:04.898590 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:04.898711 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:04.898718 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.899719 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.899723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:04.900839 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:04.900850 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.901522 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.901526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:04.901701 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:04.902181 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:04.902198 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:04.904097 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:04.907101 (+ 3004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:04.907105 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:04.907107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:04.907532 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:04.907840 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:04.907843 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:04.907845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:04.907965 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:04.908388 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:04.908391 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:04.908509 (+ 118us) client_negotiation.cc:769] Sending connection context
0929 20:17:04.908615 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 20:17:04.908813 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327}
I20260929 20:17:04.909081 2778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:04.897591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34280 (local address 127.27.185.125:46841)
0929 20:17:04.897784 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:04.897789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:04.898008 (+ 219us) server_negotiation.cc:407] Connection header received
0929 20:17:04.898133 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:04.898137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:04.898199 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:04.898276 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:04.898827 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.899579 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.900982 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.901403 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.901503 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:04.904264 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:04.904268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:04.904272 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:04.904303 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:04.906989 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:04.907626 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:04.907630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:04.907632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:04.907749 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:04.908050 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:04.908053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:04.908055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:04.908295 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:04.908401 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:04.908691 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 20:17:04.908855 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 20:17:04.910240 2818 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.124" port: 42155 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:17:04 dist-test-slave-4cx3 krb5kdc[2056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713024, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.124@KRBTEST.COM for oryx/127.27.185.124@KRBTEST.COM
I20260929 20:17:04.921717 2822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:04.910890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:42155 (local address 127.0.0.1:54952)
0929 20:17:04.911034 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:04.911049 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:04.911122 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:04.911339 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:04.911342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:04.911714 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:04.911852 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:04.911859 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.912838 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.912842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:04.913645 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:04.913654 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.914251 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.914255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:04.914406 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:04.914889 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:04.914904 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:04.916907 (+ 2003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:04.919647 (+ 2740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:04.919650 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:04.919653 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:04.919986 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:04.920283 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:04.920286 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:04.920288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:04.920478 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:04.920907 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:04.920910 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:04.921020 (+ 110us) client_negotiation.cc:769] Sending connection context
0929 20:17:04.921139 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 20:17:04.921312 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 20:17:04.921749 2821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:04.910894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54952 (local address 127.27.185.124:42155)
0929 20:17:04.911132 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:04.911136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:04.911149 (+ 13us) server_negotiation.cc:407] Connection header received
0929 20:17:04.911187 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:04.911190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:04.911237 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:04.911308 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:04.911965 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.912735 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.913758 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.914134 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.914227 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:04.917083 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:04.917088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:04.917091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:04.917117 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:04.919537 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:04.920079 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:04.920082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:04.920084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:04.920194 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:04.920569 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:04.920573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:04.920575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:04.920812 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:04.920915 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:04.921214 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 20:17:04.921507 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":164}
I20260929 20:17:04.926482 2818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3: Bootstrap starting.
I20260929 20:17:04.929678 2818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:04.930714 2818 log.cc:824] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3: Log is configured to *not* fsync() on all Append() calls
I20260929 20:17:04.933230 2818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3: No bootstrap required, opened a new log
I20260929 20:17:04.940362 2818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 42155 } }
I20260929 20:17:04.940752 2818 raft_consensus.cc:374] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:04.940884 2818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927955a2d44a4c7198c39f371d6671d3, State: Initialized, Role: FOLLOWER
I20260929 20:17:04.941517 2818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 42155 } }
I20260929 20:17:04.942400 2824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 42155 } } }
I20260929 20:17:04.942687 2824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:17:04.943465 2818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:17:04.949910 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 2162
W20260929 20:17:04.953136 2835 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:17:04.953260 2835 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:17:04.957197 2664 master_runner.cc:428] Detected that this master 927955a2d44a4c7198c39f371d6671d3 is joining an existing cluster
I20260929 20:17:04.957316 2664 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.185.124:42155
Sep 29 20:17:04 dist-test-slave-4cx3 krb5kdc[2056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.185.65@KRBTEST.COM
I20260929 20:17:04.973624 2837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:04.951625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39464 (local address 127.27.185.65:42717)
0929 20:17:04.952210 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:04.952218 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:04.952248 (+ 30us) server_negotiation.cc:407] Connection header received
0929 20:17:04.952316 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:04.952323 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:04.952618 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:04.953972 (+ 1354us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:04.954476 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.956892 (+ 2416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.958823 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:04.959717 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:04.965255 (+ 5538us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:04.965563 (+ 308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:04.965577 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:04.965595 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:04.965632 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:04.969464 (+ 3832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:04.970305 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:04.970314 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:04.970318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:04.970484 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:04.970947 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:04.970952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:04.970955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:04.971364 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:04.972790 (+ 1426us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:04.972946 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 20:17:04.973210 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":1281,"threads_started":1}
I20260929 20:17:04.975245 2285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:05.018723 2778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:04.984497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34292 (local address 127.27.185.125:46841)
0929 20:17:04.984693 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:04.984699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:04.997312 (+ 12613us) server_negotiation.cc:407] Connection header received
0929 20:17:04.998637 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:04.998644 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:04.998735 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:04.998812 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:05.003492 (+ 4680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.004645 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.008084 (+ 3439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.008768 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.008870 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:05.011724 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:05.011731 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:05.011736 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:05.011778 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:05.015401 (+ 3623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:05.016281 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:05.016288 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:05.016292 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:05.016453 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:05.016947 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:05.016952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:05.016956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:05.017322 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:05.017456 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:05.018117 (+ 661us) server_negotiation.cc:299] Negotiation successful
0929 20:17:05.018373 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 20:17:05.020917 2844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:04.984475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:46841 (local address 127.0.0.1:34292)
0929 20:17:04.997169 (+ 12694us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:04.997190 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:04.998254 (+ 1064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:04.999080 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:04.999089 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:05.002110 (+ 3021us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:05.002532 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:05.002546 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.004878 (+ 2332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.004884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:05.006675 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:05.006689 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.009210 (+ 2521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.009216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:05.009440 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:05.010248 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:05.010277 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:05.011503 (+ 1226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:05.015575 (+ 4072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:05.015580 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:05.015584 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:05.016130 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:05.016609 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:05.016613 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:05.016617 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:05.016817 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:05.017541 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:05.017546 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:05.017793 (+ 247us) client_negotiation.cc:769] Sending connection context
0929 20:17:05.019008 (+ 1215us) client_negotiation.cc:240] Negotiation successful
0929 20:17:05.020301 (+ 1293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12516,"thread_start_us":17294,"threads_started":1}
I20260929 20:17:05.024209 2845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:04.987837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:42155 (local address 127.0.0.1:54966)
0929 20:17:05.008034 (+ 20197us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:05.008054 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:05.008226 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:05.008662 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:05.008667 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:05.009329 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:05.009802 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:05.009820 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.011335 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.011340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:05.013069 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:05.013088 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.014479 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.014486 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:05.014657 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:05.015274 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:05.015297 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:05.016743 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:05.021214 (+ 4471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:05.021217 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:05.021219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:05.021747 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:05.022174 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:05.022178 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:05.022181 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:05.022333 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:05.022989 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:05.022993 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:05.023174 (+ 181us) client_negotiation.cc:769] Sending connection context
0929 20:17:05.023327 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 20:17:05.023652 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20028,"mutex_wait_us":19,"thread_start_us":19796,"threads_started":1}
I20260929 20:17:05.025224 2821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:04.985150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54966 (local address 127.27.185.124:42155)
0929 20:17:04.985310 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:04.985316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:05.008142 (+ 22826us) server_negotiation.cc:407] Connection header received
0929 20:17:05.008337 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:05.008342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:05.008409 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:05.008489 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:05.009976 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.011116 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.013250 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.014072 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.014200 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:05.016923 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:05.016932 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:05.016938 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:05.016974 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:05.020944 (+ 3970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:05.021891 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:05.021896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:05.021899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:05.022044 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:05.022468 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:05.022474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:05.022479 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:05.022883 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:05.023031 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:05.024621 (+ 1590us) server_negotiation.cc:299] Negotiation successful
0929 20:17:05.024857 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":42}
I20260929 20:17:05.026882 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:04.984343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34516 (local address 127.27.185.126:36737)
0929 20:17:04.984543 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:04.984550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:05.001750 (+ 17200us) server_negotiation.cc:407] Connection header received
0929 20:17:05.002886 (+ 1136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:05.002893 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:05.002979 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:05.003066 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:05.008045 (+ 4979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.009241 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.011599 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.012269 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.012377 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:05.015114 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:05.015123 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:05.015131 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:05.015184 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:05.019778 (+ 4594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:05.023441 (+ 3663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:05.023447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:05.023450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:05.023630 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:05.024105 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:05.024111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:05.024114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:05.024745 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:05.024885 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:05.026344 (+ 1459us) server_negotiation.cc:299] Negotiation successful
0929 20:17:05.026626 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 20:17:05.027809 2843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:04.987790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.0.0.1:34516)
0929 20:17:05.001634 (+ 13844us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:05.001658 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:05.002727 (+ 1069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:05.003286 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:05.003291 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:05.007483 (+ 4192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:05.007864 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:05.007877 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.009461 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.009469 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:05.011374 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:05.011413 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.012698 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.012704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:05.012945 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:05.013625 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:05.013647 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:05.014874 (+ 1227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:05.019939 (+ 5065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:05.019945 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:05.019949 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:05.023282 (+ 3333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:05.023787 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:05.023791 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:05.023795 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:05.023971 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:05.024963 (+ 992us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:05.024968 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:05.025150 (+ 182us) client_negotiation.cc:769] Sending connection context
0929 20:17:05.027151 (+ 2001us) client_negotiation.cc:240] Negotiation successful
0929 20:17:05.027520 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13690,"mutex_wait_us":56,"thread_start_us":9537,"threads_started":1}
Sep 29 20:17:05 dist-test-slave-4cx3 krb5kdc[2056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713021, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.126@KRBTEST.COM for oryx/127.27.185.124@KRBTEST.COM
I20260929 20:17:05.046029 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:05.032709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:42155 (local address 127.0.0.1:54972)
0929 20:17:05.033015 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:05.033030 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:05.033163 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:05.033567 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:05.033575 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:05.033996 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:05.034153 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:05.034162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.035149 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.035155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:05.036124 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:05.036136 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.036932 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.036938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:05.037113 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:05.037724 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:05.037743 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:05.039968 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:05.043209 (+ 3241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:05.043215 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:05.043220 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:05.043756 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:05.044132 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:05.044154 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:05.044158 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:05.044312 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:05.044976 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:05.044981 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:05.045196 (+ 215us) client_negotiation.cc:769] Sending connection context
0929 20:17:05.045339 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 20:17:05.045649 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181}
I20260929 20:17:05.046032 2821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:05.032752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54972 (local address 127.27.185.124:42155)
0929 20:17:05.032918 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:05.032924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:05.033105 (+ 181us) server_negotiation.cc:407] Connection header received
0929 20:17:05.033328 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:05.033333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:05.033426 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:05.033546 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:05.034297 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.035013 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.036276 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.036766 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.036899 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:05.040204 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:05.040208 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:05.040212 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:05.040243 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:05.043020 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:05.043866 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:05.043871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:05.043875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:05.044015 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:05.044469 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:05.044473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:05.044475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:05.044843 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:05.045022 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:05.045464 (+ 442us) server_negotiation.cc:299] Negotiation successful
0929 20:17:05.045691 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:17:05.047886 2101 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.185.124:42155: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "927955a2d44a4c7198c39f371d6671d3"
member_type: NON_VOTER
last_known_addr {
host: "127.27.185.124"
port: 42155
}
attrs {
promote: true
}
}
dest_uuid: "3a60def670884f8886ada0b1616e4159"
cas_config_opid_index: 5
I20260929 20:17:05.048803 2101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: NON_VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: true } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: NON_VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: true } }
I20260929 20:17:05.050731 2662 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 927955a2d44a4c7198c39f371d6671d3 (127.27.185.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: NON_VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: true } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: NON_VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: true } } }
I20260929 20:17:05.056520 2711 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [term 1 LEARNER]: Refusing update from remote peer 3a60def670884f8886ada0b1616e4159: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 20:17:05.058584 2793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:17:05.063866 2662 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.185.124:42155
I20260929 20:17:05.065090 2793 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [term 1 FOLLOWER]: Refusing update from remote peer 3a60def670884f8886ada0b1616e4159: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 20:17:05.085398 2848 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: NON_VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260929 20:17:05.093386 2771 tablet.cc:2369] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:17:05.099850 2664 master.cc:537] Master@127.27.185.124:42155 shutting down...
I20260929 20:17:05.114440 2664 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:17:05.114888 2664 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:17:05.115098 2664 tablet_replica.cc:330] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3: stopping tablet replica
I20260929 20:17:05.123231 2664 master.cc:552] Master@127.27.185.124:42155 shutdown complete.
I20260929 20:17:05.123322 2664 master_runner.cc:305] Clearing existing system tablet
I20260929 20:17:05.123720 2664 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:05.123840 2664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:05.129689 2711 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 927955a2d44a4c7198c39f371d6671d3 (127.27.185.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: NON_VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: true } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: NON_VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: true } } }
I20260929 20:17:05.134155 2739 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a60def670884f8886ada0b1616e4159" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: NON_VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: true } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: NON_VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: true } } }
I20260929 20:17:05.134502 2739 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [sys.catalog]: This master's current role is: LEARNER
I20260929 20:17:05.135834 2846 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: NON_VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:17:05.143625 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 2162
I20260929 20:17:05.144764 2664 fs_manager.cc:709] Time spent opening directory manager: real 0.020s user 0.002s sys 0.001s
I20260929 20:17:05.149314 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:05.148678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:42155 (local address 127.0.0.1:54974)
0929 20:17:05.148950 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:05.149090 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260929 20:17:05.158855 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:05.158355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:42155 (local address 127.0.0.1:54990)
0929 20:17:05.158546 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:05.158654 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 20:17:05.159264 2086 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 -> Peer 927955a2d44a4c7198c39f371d6671d3 (127.27.185.124:42155): Couldn't send request to peer 927955a2d44a4c7198c39f371d6671d3. Status: Network error: Client connection negotiation failed: client connection to 127.27.185.124:42155: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 20:17:05.161994 2662 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159: attempting to promote NON_VOTER ceb7a8c34c984b92ab34d3a2e1d14d49 to VOTER
I20260929 20:17:05.162120 2848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3a60def670884f8886ada0b1616e4159" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: NON_VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: true } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: NON_VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: true } } }
I20260929 20:17:05.162528 2848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [sys.catalog]: This master's current role is: LEADER
I20260929 20:17:05.163048 2662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: NON_VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: true } }
I20260929 20:17:05.164208 2853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:05.165931 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:05.165447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:42155 (local address 127.0.0.1:54996)
0929 20:17:05.165655 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:05.165745 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260929 20:17:05.165907 2711 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [term 1 LEARNER]: Refusing update from remote peer 3a60def670884f8886ada0b1616e4159: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 20:17:05.166635 2846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a60def670884f8886ada0b1616e4159" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: NON_VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: true } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: NON_VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: NON_VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: true } } }
I20260929 20:17:05.167088 2846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [sys.catalog]: This master's current role is: LEADER
I20260929 20:17:05.167182 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:05.166602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:42155 (local address 127.0.0.1:55006)
0929 20:17:05.166913 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:05.167038 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260929 20:17:05.167279 2846 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260929 20:17:05.167635 2086 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 -> Peer 927955a2d44a4c7198c39f371d6671d3 (127.27.185.124:42155): Couldn't send request to peer 927955a2d44a4c7198c39f371d6671d3. Status: Network error: Client connection negotiation failed: client connection to 127.27.185.124:42155: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 20:17:05.170656 2711 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ceb7a8c34c984b92ab34d3a2e1d14d49 (127.27.185.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: NON_VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: true } } }
I20260929 20:17:05.174965 2664 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:05.175135 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:42717
--local_ip_for_outbound_sockets=127.27.185.65
--tserver_master_addrs=127.27.185.126:36737
--webserver_port=43195
--webserver_interface=127.27.185.65
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.65.keytab
--principal=oryx/127.27.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:40591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.185.126:36737,127.27.185.125:46841
--tserver_master_addrs=127.27.185.126:36737,127.27.185.125:46841,127.27.185.124:42155 with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:17:05.175273 2664 fs_manager.cc:725] Time spent opening block manager: real 0.028s user 0.000s sys 0.002s
I20260929 20:17:05.175402 2664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-2/wal
uuid: "927955a2d44a4c7198c39f371d6671d3"
format_stamp: "Formatted at 2026-09-29 20:17:04 on dist-test-slave-4cx3"
server_key: "f38f0052626cfb3e01bd9fb351609900"
server_key_iv: "a8b6bbcfb52b1eead0e74a994be3e274"
server_key_version: "encryptionkey@0"
I20260929 20:17:05.170013 2846 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ceb7a8c34c984b92ab34d3a2e1d14d49 (127.27.185.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: NON_VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: true } } }
I20260929 20:17:05.181067 2664 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:17:05.181563 2739 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a60def670884f8886ada0b1616e4159" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: NON_VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: true } } }
I20260929 20:17:05.181862 2739 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:17:05.184314 2848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3a60def670884f8886ada0b1616e4159" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: NON_VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: true } } }
I20260929 20:17:05.184607 2848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [sys.catalog]: This master's current role is: LEADER
I20260929 20:17:05.184370 2662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a60def670884f8886ada0b1616e4159" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: NON_VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: true } } }
I20260929 20:17:05.184894 2662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [sys.catalog]: This master's current role is: LEADER
I20260929 20:17:05.188262 2664 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 20:17:05.188452 2664 log.cc:1196] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 20:17:05.188866 2664 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3: Deleting consensus metadata
I20260929 20:17:05.189116 2664 master_runner.cc:315] Copying system tablet from 127.27.185.126:36737
I20260929 20:17:05.191605 2664 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3: tablet copy: Beginning tablet copy session from remote peer at address 127.27.185.126:36737
I20260929 20:17:05.205221 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:05.192360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.0.0.1:34530)
0929 20:17:05.193065 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:05.193081 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:05.193249 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:05.193651 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:05.193656 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:05.194190 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:05.194407 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:05.194416 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.195485 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.195492 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:05.196425 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:05.196436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.197167 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.197173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:05.197350 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:05.198108 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:05.198132 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:05.198874 (+ 742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:05.202528 (+ 3654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:05.202534 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:05.202538 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:05.202884 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:05.203265 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:05.203270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:05.203274 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:05.203482 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:05.204119 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:05.204125 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:05.204273 (+ 148us) client_negotiation.cc:769] Sending connection context
0929 20:17:05.204416 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 20:17:05.204734 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":447,"threads_started":1}
I20260929 20:17:05.205220 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:05.192363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34530 (local address 127.27.185.126:36737)
0929 20:17:05.192635 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:05.192644 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:05.193170 (+ 526us) server_negotiation.cc:407] Connection header received
0929 20:17:05.193378 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:05.193384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:05.193469 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:05.193583 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:05.194551 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.195263 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.196582 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.197005 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.197143 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:05.199094 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:05.199099 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:05.199102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:05.199133 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:05.202394 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:05.203011 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:05.203016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:05.203020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:05.203155 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:05.203598 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:05.203604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:05.203608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:05.203872 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:05.204039 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:05.204507 (+ 468us) server_negotiation.cc:299] Negotiation successful
0929 20:17:05.204725 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260929 20:17:05.206287 2121 tablet_copy_service.cc:140] P 3a60def670884f8886ada0b1616e4159: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 927955a2d44a4c7198c39f371d6671d3 ({username='oryx', principal='oryx/127.27.185.124@KRBTEST.COM'} at 127.0.0.1:34530)
I20260929 20:17:05.206454 2121 tablet_copy_service.cc:161] P 3a60def670884f8886ada0b1616e4159: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 927955a2d44a4c7198c39f371d6671d3 at {username='oryx', principal='oryx/127.27.185.124@KRBTEST.COM'} at 127.0.0.1:34530: session id = 927955a2d44a4c7198c39f371d6671d3-00000000000000000000000000000000
I20260929 20:17:05.209136 2121 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 20:17:05.210928 2664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 20:17:05.216040 2664 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3: tablet copy: Starting download of 0 data blocks...
I20260929 20:17:05.216327 2664 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3: tablet copy: Starting download of 1 WAL segments...
I20260929 20:17:05.218616 2664 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 20:17:05.222159 2121 tablet_copy_service.cc:342] P 3a60def670884f8886ada0b1616e4159: Request end of tablet copy session 927955a2d44a4c7198c39f371d6671d3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.185.124@KRBTEST.COM'} at 127.0.0.1:34530
I20260929 20:17:05.222342 2121 tablet_copy_service.cc:433] P 3a60def670884f8886ada0b1616e4159: ending tablet copy session 927955a2d44a4c7198c39f371d6671d3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 927955a2d44a4c7198c39f371d6671d3
I20260929 20:17:05.232394 2664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:05.235168 2862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:05.235958 2863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:05.244118 2865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:05.244711 2664 server_base.cc:1034] running on GCE node
I20260929 20:17:05.245059 2664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:05.245745 2664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:05.246937 2664 hybrid_clock.cc:648] HybridClock initialized: now 1790713025246894 us; error 33 us; skew 500 ppm
I20260929 20:17:05.247191 2664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:17:05 dist-test-slave-4cx3 krb5kdc[2056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713025, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:17:05.249286 2664 init.cc:377] Logged in from keytab as oryx/127.27.185.124@KRBTEST.COM (short username oryx)
I20260929 20:17:05.250522 2664 webserver.cc:460] Webserver started at http://127.27.185.124:42977/ using document root <none> and password file <none>
I20260929 20:17:05.250821 2664 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:05.250933 2664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:05.253270 2664 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260929 20:17:05.254905 2872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:05.255321 2664 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260929 20:17:05.255505 2664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-2/wal
uuid: "927955a2d44a4c7198c39f371d6671d3"
format_stamp: "Formatted at 2026-09-29 20:17:04 on dist-test-slave-4cx3"
server_key: "f38f0052626cfb3e01bd9fb351609900"
server_key_iv: "a8b6bbcfb52b1eead0e74a994be3e274"
server_key_version: "encryptionkey@0"
I20260929 20:17:05.255705 2664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:05.295877 2664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:05.299312 2664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:05.358083 2664 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.124:42155
I20260929 20:17:05.358131 2923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.124:42155 every 8 connection(s)
I20260929 20:17:05.359127 2664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/master-2/data/info.pb
I20260929 20:17:05.366463 2924 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:17:05.373167 2924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3: Bootstrap starting.
I20260929 20:17:05.388418 2924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:17:05.388892 2924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3: Bootstrap complete.
I20260929 20:17:05.389827 2924 raft_consensus.cc:348] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: NON_VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: true } }
I20260929 20:17:05.390090 2924 raft_consensus.cc:729] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 927955a2d44a4c7198c39f371d6671d3, State: Initialized, Role: LEARNER
I20260929 20:17:05.390323 2924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: NON_VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: true } }
I20260929 20:17:05.391147 2927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: NON_VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: true } } }
I20260929 20:17:05.391461 2924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:17:05.391621 2927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [sys.catalog]: This master's current role is: LEARNER
I20260929 20:17:05.401474 2938 catalog_manager.cc:1259] Loaded cluster ID: 09b63d71bcc342fcaf7a45474598022f
I20260929 20:17:05.401597 2938 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3: loading cluster ID for follower catalog manager: success
I20260929 20:17:05.404223 2938 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3: acquiring CA information for follower catalog manager: success
I20260929 20:17:05.406247 2938 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 20:17:05.611462 2855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:05.611733 2855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:05.611918 2855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:05.619477 2855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:05.619585 2855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:05.619765 2855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:17:05.628266 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:05.621296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:42155 (local address 127.0.0.1:55020)
0929 20:17:05.621534 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:05.621557 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:05.621700 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:05.622554 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:05.622558 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:05.622584 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:17:05.622748 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:05.622755 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.624614 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.624619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:05.625959 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:05.625974 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.627640 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.627645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:05.627792 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:05.627820 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 20:17:05.627913 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 20:17:05.628046 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 20:17:05.628403 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:05.621528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55020 (local address 127.27.185.124:42155)
0929 20:17:05.622186 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:05.622192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:05.622213 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:17:05.622291 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:05.622296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:05.622388 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:05.622534 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:17:05.622907 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.624431 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.626122 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.627444 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.627640 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:05.627829 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:05.627945 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 20:17:05.628080 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":365,"threads_started":1}
I20260929 20:17:05.629261 2899 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 20:17:05.629957 2927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a60def670884f8886ada0b1616e4159. Latest consensus state: current_term: 1 leader_uuid: "3a60def670884f8886ada0b1616e4159" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: NON_VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: true } } }
I20260929 20:17:05.630260 2927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [sys.catalog]: This master's current role is: LEARNER
I20260929 20:17:05.632144 2855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:40591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.65.keytab
--principal=oryx/127.27.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:42717
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=43195
--tserver_master_addrs=127.27.185.126:36737,127.27.185.125:46841,127.27.185.124:42155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:05.633512 2855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:05.634859 2855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:05.648365 2947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:05.648419 2945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:05.648429 2944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:05.649293 2855 server_base.cc:1034] running on GCE node
I20260929 20:17:05.649924 2855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:05.650921 2855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:05.652151 2855 hybrid_clock.cc:648] HybridClock initialized: now 1790713025652095 us; error 47 us; skew 500 ppm
I20260929 20:17:05.652513 2855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:17:05 dist-test-slave-4cx3 krb5kdc[2056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713025, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:17:05.655824 2855 init.cc:377] Logged in from keytab as oryx/127.27.185.65@KRBTEST.COM (short username oryx)
I20260929 20:17:05.657326 2855 webserver.cc:460] Webserver started at http://127.27.185.65:43195/ using document root <none> and password file <none>
I20260929 20:17:05.657999 2855 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:05.658154 2855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:05.663743 2855 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 20:17:05.666617 2954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:05.668030 2855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:17:05.668213 2855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "eaa3fccfe54b400e80a3f4c5c33a1baf"
format_stamp: "Formatted at 2026-09-29 20:17:02 on dist-test-slave-4cx3"
server_key: "bfe614719920366408765a5601d64f33"
server_key_iv: "d209c48deea247af2ebd40a555b40a57"
server_key_version: "encryptionkey@0"
I20260929 20:17:05.669251 2855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:05.708756 2855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:05.713160 2855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:05.713480 2855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:05.714493 2855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:17:05.716012 2855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:17:05.716136 2855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:05.716274 2855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:17:05.716370 2855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:05.750903 2855 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:42717
I20260929 20:17:05.750998 3066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:42717 every 8 connection(s)
I20260929 20:17:05.752487 2855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:17:05.756992 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 2855
I20260929 20:17:05.758016 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 2296
Sep 29 20:17:05 dist-test-slave-4cx3 krb5kdc[2056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.185.66@KRBTEST.COM
Sep 29 20:17:05 dist-test-slave-4cx3 krb5kdc[2056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713025, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.125@KRBTEST.COM
Sep 29 20:17:05 dist-test-slave-4cx3 krb5kdc[2056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713025, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM
Sep 29 20:17:05 dist-test-slave-4cx3 krb5kdc[2056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713025, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.124@KRBTEST.COM
I20260929 20:17:05.793380 3077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:05.759695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39518 (local address 127.27.185.66:33241)
0929 20:17:05.769173 (+ 9478us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:05.769185 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:05.770053 (+ 868us) server_negotiation.cc:407] Connection header received
0929 20:17:05.770153 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:05.770162 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:05.770450 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:05.770666 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:05.773290 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.775206 (+ 1916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.776950 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.777716 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.777857 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:05.784173 (+ 6316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:05.784188 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:05.784205 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:05.784250 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:05.788631 (+ 4381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:05.789637 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:05.789649 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:05.789653 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:05.789823 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:05.790304 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:05.790309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:05.790312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:05.790764 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:05.792249 (+ 1485us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:05.792468 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 20:17:05.792756 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9310,"thread_start_us":9168,"threads_started":1}
I20260929 20:17:05.794937 2419 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:05.795624 3072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:05.755801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:43577 (local address 127.27.185.126:36737)
0929 20:17:05.763878 (+ 8077us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:05.763888 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:05.768286 (+ 4398us) server_negotiation.cc:407] Connection header received
0929 20:17:05.772076 (+ 3790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:05.772081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:05.772180 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:05.772313 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:05.775936 (+ 3623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.777086 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.779827 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.780501 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.780645 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:05.787977 (+ 7332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:05.787996 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:05.788002 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:05.788053 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:05.791604 (+ 3551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:05.792730 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:05.792735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:05.792738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:05.792880 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:05.793562 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:05.793568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:05.793571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:05.793913 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:05.794052 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:05.794996 (+ 944us) server_negotiation.cc:299] Negotiation successful
0929 20:17:05.795214 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7899,"thread_start_us":7776,"threads_started":1}
I20260929 20:17:05.796439 3071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:05.755473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:49447 (local address 127.27.185.125:46841)
0929 20:17:05.764250 (+ 8777us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:05.764255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:05.773548 (+ 9293us) server_negotiation.cc:407] Connection header received
0929 20:17:05.773619 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:05.773624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:05.773715 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:05.773837 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:05.777339 (+ 3502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.778479 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.781862 (+ 3383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.782313 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.782459 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:05.787048 (+ 4589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:05.787054 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:05.787059 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:05.787108 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:05.791593 (+ 4485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:05.792815 (+ 1222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:05.792833 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:05.792837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:05.792994 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:05.793727 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:05.793732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:05.793738 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:05.794167 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:05.794299 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:05.795120 (+ 821us) server_negotiation.cc:299] Negotiation successful
0929 20:17:05.795499 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8666,"thread_start_us":8566,"threads_started":1}
I20260929 20:17:05.796901 3075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:05.756338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.27.185.65:43577)
0929 20:17:05.768104 (+ 11766us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:05.768152 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:05.771692 (+ 3540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:05.773566 (+ 1874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:05.773580 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:05.774647 (+ 1067us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:05.775715 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:05.775742 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.777249 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.777254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:05.779669 (+ 2415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:05.779683 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.780763 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.780772 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:05.781116 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:05.782957 (+ 1841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:05.782984 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:05.787703 (+ 4719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:05.791769 (+ 4066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:05.791778 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:05.791849 (+ 71us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:05.792591 (+ 742us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:05.793094 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:05.793099 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:05.793103 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:05.793414 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:05.794138 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:05.794146 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:05.794636 (+ 490us) client_negotiation.cc:769] Sending connection context
0929 20:17:05.794904 (+ 268us) client_negotiation.cc:240] Negotiation successful
0929 20:17:05.795533 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11433,"mutex_wait_us":128,"spinlock_wait_cycles":32256,"thread_start_us":15719,"threads_started":1}
I20260929 20:17:05.798010 3074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:05.758577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:46841 (local address 127.27.185.65:49447)
0929 20:17:05.773302 (+ 14725us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:05.773321 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:05.773436 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:05.774052 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:05.774057 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:05.774814 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:05.775715 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:05.775742 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.778622 (+ 2880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.778626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:05.781737 (+ 3111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:05.781751 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.782511 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.782517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:05.782697 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:05.783487 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:05.783513 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:05.786771 (+ 3258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:05.791771 (+ 5000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:05.791781 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:05.791799 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:05.792611 (+ 812us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:05.793123 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:05.793130 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:05.793135 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:05.793349 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:05.794352 (+ 1003us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:05.794356 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:05.794574 (+ 218us) client_negotiation.cc:769] Sending connection context
0929 20:17:05.796730 (+ 2156us) client_negotiation.cc:240] Negotiation successful
0929 20:17:05.797407 (+ 677us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14603,"mutex_wait_us":205,"thread_start_us":9088,"threads_started":1}
I20260929 20:17:05.798662 3068 heartbeater.cc:347] Connected to a master server at 127.27.185.126:36737
I20260929 20:17:05.799002 3069 heartbeater.cc:347] Connected to a master server at 127.27.185.125:46841
I20260929 20:17:05.799155 3068 heartbeater.cc:464] Registering TS with master...
I20260929 20:17:05.799157 3069 heartbeater.cc:464] Registering TS with master...
I20260929 20:17:05.800242 3068 heartbeater.cc:511] Master 127.27.185.126:36737 requested a full tablet report, sending...
I20260929 20:17:05.800268 3069 heartbeater.cc:511] Master 127.27.185.125:46841 requested a full tablet report, sending...
I20260929 20:17:05.801491 3073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:05.757716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:42155 (local address 127.27.185.65:44191)
0929 20:17:05.772669 (+ 14953us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:05.772693 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:05.773356 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:05.773949 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:05.773954 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:05.774842 (+ 888us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:05.777154 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:05.777169 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.778618 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.778623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:05.780548 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:05.780564 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.783992 (+ 3428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.783999 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:05.784190 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:05.785121 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:05.785144 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:05.788378 (+ 3234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:05.792367 (+ 3989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:05.792371 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:05.792374 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:05.792908 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:05.793361 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:05.793366 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:05.793370 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:05.793551 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:05.794250 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:05.794255 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:05.800604 (+ 6349us) client_negotiation.cc:769] Sending connection context
0929 20:17:05.800783 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 20:17:05.801062 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14767,"mutex_wait_us":7914,"thread_start_us":15274,"threads_started":1}
I20260929 20:17:05.801962 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:05.755478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:44191 (local address 127.27.185.124:42155)
0929 20:17:05.755688 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:05.755694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:05.772807 (+ 17113us) server_negotiation.cc:407] Connection header received
0929 20:17:05.773577 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:05.773583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:05.773671 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:05.773742 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:05.777327 (+ 3585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.778415 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.781784 (+ 3369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:05.782533 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:05.782661 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:05.788613 (+ 5952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:05.788620 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:05.788625 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:05.788676 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:05.792198 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:05.793044 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:05.793051 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:05.793058 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:05.793205 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:05.793661 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:05.793666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:05.793670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:05.794038 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:05.794167 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:05.801029 (+ 6862us) server_negotiation.cc:299] Negotiation successful
0929 20:17:05.801310 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:17:05.802186 2101 ts_manager.cc:194] Re-registered known tserver with Master: eaa3fccfe54b400e80a3f4c5c33a1baf (127.27.185.65:42717)
I20260929 20:17:05.802897 2701 ts_manager.cc:194] Registered new tserver with Master: eaa3fccfe54b400e80a3f4c5c33a1baf (127.27.185.65:42717)
I20260929 20:17:05.803025 3067 heartbeater.cc:347] Connected to a master server at 127.27.185.124:42155
I20260929 20:17:05.803126 3067 heartbeater.cc:464] Registering TS with master...
I20260929 20:17:05.803275 2101 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.185.65@KRBTEST.COM'} at 127.27.185.65:43577
I20260929 20:17:05.803539 3067 heartbeater.cc:511] Master 127.27.185.124:42155 requested a full tablet report, sending...
I20260929 20:17:05.805313 2889 ts_manager.cc:194] Registered new tserver with Master: eaa3fccfe54b400e80a3f4c5c33a1baf (127.27.185.65:42717)
I20260929 20:17:05.891336 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 2296
I20260929 20:17:05.909029 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:33241
--local_ip_for_outbound_sockets=127.27.185.66
--tserver_master_addrs=127.27.185.126:36737
--webserver_port=35653
--webserver_interface=127.27.185.66
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.66.keytab
--principal=oryx/127.27.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:40591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.185.126:36737,127.27.185.125:46841
--tserver_master_addrs=127.27.185.126:36737,127.27.185.125:46841,127.27.185.124:42155 with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:17:06.026855 2846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a60def670884f8886ada0b1616e4159" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: NON_VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: true } } }
I20260929 20:17:06.027207 2846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [sys.catalog]: This master's current role is: LEADER
I20260929 20:17:06.028172 2846 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159: attempting to promote NON_VOTER 927955a2d44a4c7198c39f371d6671d3 to VOTER
I20260929 20:17:06.028746 2846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: false } }
I20260929 20:17:06.030553 2899 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [term 1 LEARNER]: Refusing update from remote peer 3a60def670884f8886ada0b1616e4159: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 20:17:06.030979 2711 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [term 1 FOLLOWER]: Refusing update from remote peer 3a60def670884f8886ada0b1616e4159: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 20:17:06.031122 3080 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 20:17:06.031639 2846 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 20:17:06.034268 3081 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 927955a2d44a4c7198c39f371d6671d3 (127.27.185.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: false } } }
I20260929 20:17:06.034891 2711 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 927955a2d44a4c7198c39f371d6671d3 (127.27.185.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: false } } }
I20260929 20:17:06.036015 2899 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 927955a2d44a4c7198c39f371d6671d3 (127.27.185.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: false } } }
I20260929 20:17:06.039193 3084 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a60def670884f8886ada0b1616e4159" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: false } } }
I20260929 20:17:06.039667 3084 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:17:06.040271 2927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a60def670884f8886ada0b1616e4159" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: false } } }
I20260929 20:17:06.040522 2927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:17:06.044655 2846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3a60def670884f8886ada0b1616e4159" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: false } } }
I20260929 20:17:06.044626 3080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a60def670884f8886ada0b1616e4159" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: false } } }
I20260929 20:17:06.044695 3081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a60def670884f8886ada0b1616e4159" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: false } } }
I20260929 20:17:06.044988 3080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [sys.catalog]: This master's current role is: LEADER
I20260929 20:17:06.045058 3081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [sys.catalog]: This master's current role is: LEADER
I20260929 20:17:06.045156 2846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159 [sys.catalog]: This master's current role is: LEADER
W20260929 20:17:06.262171 3079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:06.262470 3079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:06.262657 3079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:06.270200 3079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:06.270354 3079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:06.270521 3079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:17:06.281008 3079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:40591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.66.keytab
--principal=oryx/127.27.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:33241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=35653
--tserver_master_addrs=127.27.185.126:36737,127.27.185.125:46841,127.27.185.124:42155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:06.282400 3079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:06.283854 3079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:06.297894 3090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:06.297925 3091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:06.298432 3093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:06.298556 3079 server_base.cc:1034] running on GCE node
I20260929 20:17:06.299453 3079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:06.300503 3079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:06.301699 3079 hybrid_clock.cc:648] HybridClock initialized: now 1790713026301635 us; error 64 us; skew 500 ppm
I20260929 20:17:06.301975 3079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:17:06 dist-test-slave-4cx3 krb5kdc[2056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713026, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:17:06.305685 3079 init.cc:377] Logged in from keytab as oryx/127.27.185.66@KRBTEST.COM (short username oryx)
I20260929 20:17:06.307200 3079 webserver.cc:460] Webserver started at http://127.27.185.66:35653/ using document root <none> and password file <none>
I20260929 20:17:06.307955 3079 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:06.308115 3079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:06.313951 3079 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:17:06.316620 3100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:06.318073 3079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:17:06.318207 3079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "100246ec987b437eb04e30cf2815bced"
format_stamp: "Formatted at 2026-09-29 20:17:03 on dist-test-slave-4cx3"
server_key: "0adb6320d9ce3a8a945349515a13ca79"
server_key_iv: "fc12aa43537c47c1596ef9a7e0b4a646"
server_key_version: "encryptionkey@0"
I20260929 20:17:06.319339 3079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:06.360589 3079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:06.365252 3079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:06.365582 3079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:06.366616 3079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:17:06.368119 3079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:17:06.368245 3079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:06.368381 3079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:17:06.368472 3079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:06.401360 3079 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:33241
I20260929 20:17:06.401446 3212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:33241 every 8 connection(s)
I20260929 20:17:06.402849 3079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:17:06.405542 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 3079
I20260929 20:17:06.405833 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 2430
Sep 29 20:17:06 dist-test-slave-4cx3 krb5kdc[2056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.185.67@KRBTEST.COM
Sep 29 20:17:06 dist-test-slave-4cx3 krb5kdc[2056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713026, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM
Sep 29 20:17:06 dist-test-slave-4cx3 krb5kdc[2056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713026, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.124@KRBTEST.COM
Sep 29 20:17:06 dist-test-slave-4cx3 krb5kdc[2056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713026, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.125@KRBTEST.COM
I20260929 20:17:06.451800 3220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:06.407206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41830 (local address 127.27.185.67:45841)
0929 20:17:06.423060 (+ 15854us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:06.423076 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:06.423127 (+ 51us) server_negotiation.cc:407] Connection header received
0929 20:17:06.423236 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:06.423250 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:06.423663 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:06.427259 (+ 3596us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:06.427634 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:06.429738 (+ 2104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:06.432401 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:06.433609 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:06.434430 (+ 821us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:06.440544 (+ 6114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:06.440563 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:06.440582 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:06.440636 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:06.446208 (+ 5572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:06.447221 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:06.447233 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:06.447236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:06.447422 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:06.447882 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:06.447886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:06.447888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:06.448232 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:06.448558 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:06.450953 (+ 2395us) server_negotiation.cc:299] Negotiation successful
0929 20:17:06.451283 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15568,"thread_start_us":16658,"threads_started":1}
I20260929 20:17:06.453707 2553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:06.458148 3222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:06.407753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:42939 (local address 127.27.185.125:46841)
0929 20:17:06.434136 (+ 26383us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:06.434141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:06.434159 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:17:06.435192 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:06.435197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:06.435282 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:06.435413 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:06.438364 (+ 2951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:06.439753 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:06.442768 (+ 3015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:06.443504 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:06.443645 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:06.450288 (+ 6643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:06.450295 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:06.450301 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:06.450364 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:06.454061 (+ 3697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:06.454825 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:06.454832 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:06.454835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:06.455026 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:06.455449 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:06.455456 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:06.455459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:06.455948 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:06.456105 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:06.457539 (+ 1434us) server_negotiation.cc:299] Negotiation successful
0929 20:17:06.457796 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26244,"thread_start_us":26133,"threads_started":1}
I20260929 20:17:06.458199 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:06.406040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:33765 (local address 127.27.185.126:36737)
0929 20:17:06.406578 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:06.406584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:06.428763 (+ 22179us) server_negotiation.cc:407] Connection header received
0929 20:17:06.435073 (+ 6310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:06.435079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:06.435205 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:06.435308 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:06.438360 (+ 3052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:06.439566 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:06.442727 (+ 3161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:06.443443 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:06.443558 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:06.448961 (+ 5403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:06.448969 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:06.448974 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:06.449019 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:06.453388 (+ 4369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:06.454371 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:06.454388 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:06.454394 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:06.454599 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:06.455127 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:06.455134 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:06.455140 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:06.455627 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:06.455798 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:06.457325 (+ 1527us) server_negotiation.cc:299] Negotiation successful
0929 20:17:06.457615 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":240,"threads_started":1}
I20260929 20:17:06.459563 3219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:06.406771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.27.185.66:33765)
0929 20:17:06.428534 (+ 21763us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:06.428616 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:06.434764 (+ 6148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:06.435812 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:06.435821 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:06.436968 (+ 1147us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:06.438093 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:06.438129 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:06.439788 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:06.439794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:06.442563 (+ 2769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:06.442573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:06.443945 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:06.443953 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:06.444362 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:06.445436 (+ 1074us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:06.445493 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:06.448704 (+ 3211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:06.453561 (+ 4857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:06.453575 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:06.453594 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:06.454223 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:06.454744 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:06.454748 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:06.454752 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:06.455006 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:06.456115 (+ 1109us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:06.456125 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:06.456775 (+ 650us) client_negotiation.cc:769] Sending connection context
0929 20:17:06.457161 (+ 386us) client_negotiation.cc:240] Negotiation successful
0929 20:17:06.457830 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21324,"mutex_wait_us":692,"spinlock_wait_cycles":41472,"thread_start_us":25083,"threads_started":1}
I20260929 20:17:06.459579 3221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:06.407545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:46841 (local address 127.27.185.66:42939)
0929 20:17:06.433515 (+ 25970us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:06.433540 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:06.434783 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:06.435816 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:06.435825 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:06.436965 (+ 1140us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:06.438105 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:06.438130 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:06.439944 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:06.439955 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:06.442578 (+ 2623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:06.442595 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:06.444014 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:06.444023 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:06.444354 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:06.445130 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:06.445166 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:06.450057 (+ 4891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:06.454223 (+ 4166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:06.454228 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:06.454231 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:06.454650 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:06.455155 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:06.455158 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:06.455160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:06.455262 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:06.456166 (+ 904us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:06.456169 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:06.456768 (+ 599us) client_negotiation.cc:769] Sending connection context
0929 20:17:06.457161 (+ 393us) client_negotiation.cc:240] Negotiation successful
0929 20:17:06.458639 (+ 1478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25808,"mutex_wait_us":1531,"spinlock_wait_cycles":47872,"thread_start_us":25596,"threads_started":1}
I20260929 20:17:06.461239 3215 heartbeater.cc:347] Connected to a master server at 127.27.185.125:46841
I20260929 20:17:06.461215 3214 heartbeater.cc:347] Connected to a master server at 127.27.185.126:36737
I20260929 20:17:06.461656 3215 heartbeater.cc:464] Registering TS with master...
I20260929 20:17:06.461653 3214 heartbeater.cc:464] Registering TS with master...
I20260929 20:17:06.462711 3223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:06.408075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:43631 (local address 127.27.185.124:42155)
0929 20:17:06.431913 (+ 23838us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:06.431923 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:06.434198 (+ 2275us) server_negotiation.cc:407] Connection header received
0929 20:17:06.435268 (+ 1070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:06.435275 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:06.435359 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:06.435457 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:06.438358 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:06.439588 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:06.442726 (+ 3138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:06.443460 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:06.443557 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:06.449523 (+ 5966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:06.449528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:06.449531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:06.449567 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:06.453943 (+ 4376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:06.456070 (+ 2127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:06.456079 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:06.456086 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:06.456253 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:06.460917 (+ 4664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:06.460923 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:06.460927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:06.461323 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:06.461455 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:06.462116 (+ 661us) server_negotiation.cc:299] Negotiation successful
0929 20:17:06.462346 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23654,"thread_start_us":25598,"threads_started":1}
I20260929 20:17:06.462764 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:06.408123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:42155 (local address 127.27.185.66:43631)
0929 20:17:06.433758 (+ 25635us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:06.433777 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:06.435118 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:06.436158 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:06.436162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:06.436967 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:06.438094 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:06.438129 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:06.439788 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:06.439794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:06.442545 (+ 2751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:06.442562 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:06.443945 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:06.443953 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:06.444362 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:06.445368 (+ 1006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:06.445404 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:06.449270 (+ 3866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:06.455490 (+ 6220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:06.455495 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:06.455501 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:06.455962 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:06.460220 (+ 4258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:06.460224 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:06.460227 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:06.460423 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:06.461608 (+ 1185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:06.461616 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:06.461810 (+ 194us) client_negotiation.cc:769] Sending connection context
0929 20:17:06.461984 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 20:17:06.462285 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25536,"mutex_wait_us":580,"spinlock_wait_cycles":18176,"thread_start_us":25768,"threads_started":1}
I20260929 20:17:06.462809 3214 heartbeater.cc:511] Master 127.27.185.126:36737 requested a full tablet report, sending...
I20260929 20:17:06.462841 3215 heartbeater.cc:511] Master 127.27.185.125:46841 requested a full tablet report, sending...
I20260929 20:17:06.463963 3213 heartbeater.cc:347] Connected to a master server at 127.27.185.124:42155
I20260929 20:17:06.464092 3213 heartbeater.cc:464] Registering TS with master...
I20260929 20:17:06.464473 3213 heartbeater.cc:511] Master 127.27.185.124:42155 requested a full tablet report, sending...
I20260929 20:17:06.464709 2101 ts_manager.cc:194] Re-registered known tserver with Master: 100246ec987b437eb04e30cf2815bced (127.27.185.66:33241)
I20260929 20:17:06.465580 2889 ts_manager.cc:194] Registered new tserver with Master: 100246ec987b437eb04e30cf2815bced (127.27.185.66:33241)
I20260929 20:17:06.465855 2101 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.185.66@KRBTEST.COM'} at 127.27.185.66:33765
I20260929 20:17:06.466658 2701 ts_manager.cc:194] Registered new tserver with Master: 100246ec987b437eb04e30cf2815bced (127.27.185.66:33241)
I20260929 20:17:06.550431 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 2430
I20260929 20:17:06.567830 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:45841
--local_ip_for_outbound_sockets=127.27.185.67
--tserver_master_addrs=127.27.185.126:36737
--webserver_port=42741
--webserver_interface=127.27.185.67
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.67.keytab
--principal=oryx/127.27.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:40591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.185.126:36737,127.27.185.125:46841
--tserver_master_addrs=127.27.185.126:36737,127.27.185.125:46841,127.27.185.124:42155 with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:17:06.806257 3068 heartbeater.cc:503] Master 127.27.185.126:36737 was elected leader, sending a full tablet report...
W20260929 20:17:06.913215 3226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:06.913545 3226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:06.913717 3226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:06.921972 3226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:06.922132 3226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:06.922322 3226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:17:06.932492 3226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:40591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.67.keytab
--principal=oryx/127.27.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:45841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=42741
--tserver_master_addrs=127.27.185.126:36737,127.27.185.125:46841,127.27.185.124:42155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:06.933897 3226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:06.935258 3226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:06.949364 3231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:06.949473 3232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:06.949633 3234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:06.950484 3226 server_base.cc:1034] running on GCE node
I20260929 20:17:06.951341 3226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:06.952414 3226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:06.953657 3226 hybrid_clock.cc:648] HybridClock initialized: now 1790713026953606 us; error 49 us; skew 500 ppm
I20260929 20:17:06.954015 3226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:17:06 dist-test-slave-4cx3 krb5kdc[2056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713026, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:17:06.957710 3226 init.cc:377] Logged in from keytab as oryx/127.27.185.67@KRBTEST.COM (short username oryx)
I20260929 20:17:06.959609 3226 webserver.cc:460] Webserver started at http://127.27.185.67:42741/ using document root <none> and password file <none>
I20260929 20:17:06.960458 3226 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:06.960606 3226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:06.967764 3226 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.004s
I20260929 20:17:06.970675 3241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:06.972196 3226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:17:06.972391 3226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "67b9089a4cb642d682558e47961dcc63"
format_stamp: "Formatted at 2026-09-29 20:17:03 on dist-test-slave-4cx3"
server_key: "9238fd013f7fdd5ee461b2c913b29181"
server_key_iv: "fb5aa22b9b969e442e0a7f1245aa5fb6"
server_key_version: "encryptionkey@0"
I20260929 20:17:06.973421 3226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:06.989848 3226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:06.994288 3226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:06.994530 3226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:06.995604 3226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:17:06.997059 3226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:17:06.997148 3226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:06.997224 3226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:17:06.997282 3226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:07.030221 3226 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:45841
I20260929 20:17:07.030275 3353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:45841 every 8 connection(s)
I20260929 20:17:07.031896 3226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:17:07.032328 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 3226
Sep 29 20:17:07 dist-test-slave-4cx3 krb5kdc[2056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713026, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM
Sep 29 20:17:07 dist-test-slave-4cx3 krb5kdc[2056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713026, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.125@KRBTEST.COM
Sep 29 20:17:07 dist-test-slave-4cx3 krb5kdc[2056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713026, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.124@KRBTEST.COM
I20260929 20:17:07.099524 3362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:07.036671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:35079 (local address 127.27.185.126:36737)
0929 20:17:07.060726 (+ 24055us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:07.060736 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:07.069838 (+ 9102us) server_negotiation.cc:407] Connection header received
0929 20:17:07.070007 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:07.070013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:07.070107 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:07.070206 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:07.072808 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.073959 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.076672 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.077367 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.077483 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:07.088026 (+ 10543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:07.088033 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:07.088039 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:07.088079 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:07.091474 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:07.093373 (+ 1899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:07.093379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:07.093382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:07.093561 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:07.097913 (+ 4352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:07.097918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:07.097922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:07.098317 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:07.098477 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:07.098872 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 20:17:07.099091 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23854,"thread_start_us":24089,"threads_started":1}
I20260929 20:17:07.099998 3361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:07.036045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:42155 (local address 127.27.185.67:47241)
0929 20:17:07.061696 (+ 25651us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:07.061778 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:07.065284 (+ 3506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:07.076125 (+ 10841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:07.076131 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:07.077765 (+ 1634us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:07.078049 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:07.078061 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.079562 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.079566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:07.081207 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:07.081222 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.082453 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.083248 (+ 795us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:07.083468 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:07.084041 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:07.084069 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:07.087318 (+ 3249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:07.093111 (+ 5793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:07.093115 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:07.093118 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:07.093699 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:07.094258 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:07.094267 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:07.094273 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:07.094847 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:07.096109 (+ 1262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:07.096123 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:07.096883 (+ 760us) client_negotiation.cc:769] Sending connection context
0929 20:17:07.097349 (+ 466us) client_negotiation.cc:240] Negotiation successful
0929 20:17:07.098087 (+ 738us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25170,"spinlock_wait_cycles":22912,"thread_start_us":29709,"threads_started":1}
I20260929 20:17:07.101287 3358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:07.035061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:47241 (local address 127.27.185.124:42155)
0929 20:17:07.075288 (+ 40227us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:07.075295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:07.075315 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:17:07.075402 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:07.075407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:07.075500 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:07.075608 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:07.078247 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.079410 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.081398 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.082062 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.082198 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:07.087699 (+ 5501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:07.087710 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:07.087718 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:07.087774 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:07.092919 (+ 5145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:07.093922 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:07.093929 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:07.093934 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:07.094099 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:07.095074 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:07.095081 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:07.095085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:07.095809 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:07.095997 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:07.100422 (+ 4425us) server_negotiation.cc:299] Negotiation successful
0929 20:17:07.100744 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40096,"thread_start_us":39987,"threads_started":1}
I20260929 20:17:07.101734 3364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:07.037166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:39105 (local address 127.27.185.125:46841)
0929 20:17:07.068683 (+ 31517us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:07.068693 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:07.068719 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:17:07.068800 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:07.068808 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:07.068928 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:07.069067 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:07.072380 (+ 3313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.073545 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.077384 (+ 3839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.078083 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.078181 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:07.087880 (+ 9699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:07.087887 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:07.087891 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:07.087937 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:07.093191 (+ 5254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:07.097567 (+ 4376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:07.097573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:07.097577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:07.097735 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:07.099590 (+ 1855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:07.099596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:07.099599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:07.099969 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:07.100118 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:07.101059 (+ 941us) server_negotiation.cc:299] Negotiation successful
0929 20:17:07.101300 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31350,"thread_start_us":31242,"threads_started":1}
I20260929 20:17:07.103219 3363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:07.036695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:46841 (local address 127.27.185.67:39105)
0929 20:17:07.068012 (+ 31317us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:07.068040 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:07.068199 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:07.070322 (+ 2123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:07.070337 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:07.071109 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:07.072131 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:07.072160 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.073742 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.073747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:07.077213 (+ 3466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:07.077226 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.079550 (+ 2324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.079557 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:07.079789 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:07.082687 (+ 2898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:07.082716 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:07.087337 (+ 4621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:07.095775 (+ 8438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:07.095782 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:07.095799 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:07.096380 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:07.097906 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:07.097919 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:07.097923 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:07.098098 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:07.100232 (+ 2134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:07.100236 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:07.100399 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 20:17:07.102465 (+ 2066us) client_negotiation.cc:240] Negotiation successful
0929 20:17:07.102769 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31148,"mutex_wait_us":8,"spinlock_wait_cycles":29952,"thread_start_us":30944,"threads_started":1}
I20260929 20:17:07.103587 3354 heartbeater.cc:347] Connected to a master server at 127.27.185.124:42155
I20260929 20:17:07.103941 3354 heartbeater.cc:464] Registering TS with master...
I20260929 20:17:07.104395 3356 heartbeater.cc:347] Connected to a master server at 127.27.185.125:46841
I20260929 20:17:07.104526 3356 heartbeater.cc:464] Registering TS with master...
I20260929 20:17:07.104956 3365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:07.037515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.27.185.67:35079)
0929 20:17:07.069723 (+ 32208us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:07.069738 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:07.069886 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:07.071529 (+ 1643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:07.071533 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:07.072178 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:07.072627 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:07.072639 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.074127 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.074133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:07.076527 (+ 2394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:07.076540 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.078478 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.078486 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:07.078796 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:07.080235 (+ 1439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:07.080266 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:07.087645 (+ 7379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:07.091649 (+ 4004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:07.091662 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:07.091686 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:07.092374 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:07.097048 (+ 4674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:07.097053 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:07.097057 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:07.097241 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:07.098424 (+ 1183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:07.098428 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:07.098579 (+ 151us) client_negotiation.cc:769] Sending connection context
0929 20:17:07.098744 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 20:17:07.099023 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32083,"mutex_wait_us":467,"thread_start_us":31913,"threads_started":1}
I20260929 20:17:07.105019 3354 heartbeater.cc:511] Master 127.27.185.124:42155 requested a full tablet report, sending...
I20260929 20:17:07.105063 3356 heartbeater.cc:511] Master 127.27.185.125:46841 requested a full tablet report, sending...
I20260929 20:17:07.106482 3355 heartbeater.cc:347] Connected to a master server at 127.27.185.126:36737
I20260929 20:17:07.106604 3355 heartbeater.cc:464] Registering TS with master...
I20260929 20:17:07.107024 3355 heartbeater.cc:511] Master 127.27.185.126:36737 requested a full tablet report, sending...
I20260929 20:17:07.107056 2889 ts_manager.cc:194] Registered new tserver with Master: 67b9089a4cb642d682558e47961dcc63 (127.27.185.67:45841)
I20260929 20:17:07.107120 2701 ts_manager.cc:194] Registered new tserver with Master: 67b9089a4cb642d682558e47961dcc63 (127.27.185.67:45841)
I20260929 20:17:07.108263 2101 ts_manager.cc:194] Re-registered known tserver with Master: 67b9089a4cb642d682558e47961dcc63 (127.27.185.67:45841)
I20260929 20:17:07.109797 2101 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.185.67@KRBTEST.COM'} at 127.27.185.67:35079
I20260929 20:17:07.121320 3368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:07.085413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34532 (local address 127.27.185.126:36737)
0929 20:17:07.085937 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:07.085946 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:07.087204 (+ 1258us) server_negotiation.cc:407] Connection header received
0929 20:17:07.087531 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:07.087537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:07.087621 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:07.087707 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:07.096319 (+ 8612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.097726 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.102833 (+ 5107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.103509 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.103662 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:07.114482 (+ 10820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:07.114489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:07.114494 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:07.114535 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:07.117953 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:07.119112 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:07.119118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:07.119122 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:07.119262 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:07.119762 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:07.119767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:07.119770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:07.120118 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:07.120404 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:07.120661 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 20:17:07.120898 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":779,"threads_started":1}
I20260929 20:17:07.127027 3358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:07.093686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55028 (local address 127.27.185.124:42155)
0929 20:17:07.101587 (+ 7901us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:07.101594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:07.108810 (+ 7216us) server_negotiation.cc:407] Connection header received
0929 20:17:07.108909 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:07.108915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:07.109018 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:07.109118 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:07.113818 (+ 4700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.114913 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.116752 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.117407 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.117544 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:07.120129 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:07.120136 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:07.120141 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:07.120185 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:07.123880 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:07.124802 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:07.124807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:07.124811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:07.124950 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:07.125344 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:07.125351 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:07.125355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:07.125729 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:07.126051 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:07.126382 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 20:17:07.126622 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7762,"thread_start_us":18312,"threads_started":1}
I20260929 20:17:07.128206 3370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:07.085854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34306 (local address 127.27.185.125:46841)
0929 20:17:07.090286 (+ 4432us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:07.090293 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:07.107116 (+ 16823us) server_negotiation.cc:407] Connection header received
0929 20:17:07.108306 (+ 1190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:07.108322 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:07.108406 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:07.108513 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:07.111228 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.112632 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.114500 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.115211 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.115335 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:07.118401 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:07.118410 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:07.118417 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:07.118463 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:07.124478 (+ 6015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:07.125248 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:07.125254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:07.125259 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:07.125463 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:07.125924 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:07.125929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:07.125932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:07.126283 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:07.126584 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:07.127574 (+ 990us) server_negotiation.cc:299] Negotiation successful
0929 20:17:07.127810 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4284,"thread_start_us":2991,"threads_started":1}
I20260929 20:17:07.126233 2101 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:17:07.134469 2101 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:17:07.195472 3389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:07.155535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:42717 (local address 127.0.0.1:39480)
0929 20:17:07.180613 (+ 25078us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:07.180642 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:07.180843 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:07.183090 (+ 2247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:07.183099 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:07.183137 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:17:07.183618 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:07.183633 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.187762 (+ 4129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.187768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:07.189814 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:07.189830 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.194549 (+ 4719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.194556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:07.194762 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:07.194801 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 20:17:07.194893 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 20:17:07.195049 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24857,"thread_start_us":29867,"threads_started":1}
I20260929 20:17:07.195582 3388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:07.155788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39480 (local address 127.27.185.65:42717)
0929 20:17:07.169847 (+ 14059us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:07.169855 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:07.181927 (+ 12072us) server_negotiation.cc:407] Connection header received
0929 20:17:07.182084 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:07.182097 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:07.182507 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:07.182773 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:17:07.183894 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.187548 (+ 3654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.189998 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.194008 (+ 4010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.194169 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:07.194389 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:07.195056 (+ 667us) server_negotiation.cc:299] Negotiation successful
0929 20:17:07.195256 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13897,"thread_start_us":14051,"threads_started":1}
I20260929 20:17:07.195665 3385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:07.155150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:45841 (local address 127.0.0.1:41844)
0929 20:17:07.163140 (+ 7990us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:07.163156 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:07.163261 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:07.189014 (+ 25753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:07.189024 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:07.189060 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:17:07.189553 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:07.189564 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.191051 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.191058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:07.192842 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:07.192857 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.194772 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.194781 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:07.194982 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:07.195014 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 20:17:07.195111 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 20:17:07.195233 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7865,"mutex_wait_us":3,"thread_start_us":7684,"threads_started":1}
I20260929 20:17:07.196379 3386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:07.155457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41844 (local address 127.27.185.67:45841)
0929 20:17:07.188404 (+ 32947us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:07.188414 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:07.188448 (+ 34us) server_negotiation.cc:407] Connection header received
0929 20:17:07.188545 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:07.188552 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:07.188792 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:07.189011 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:17:07.189778 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.190898 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.193006 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.194548 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.194678 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:07.194885 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:07.195845 (+ 960us) server_negotiation.cc:299] Negotiation successful
0929 20:17:07.196125 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32766,"thread_start_us":32595,"threads_started":1}
I20260929 20:17:07.196879 3387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:07.157756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:33241 (local address 127.0.0.1:39524)
0929 20:17:07.181591 (+ 23835us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:07.181613 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:07.181749 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:07.188637 (+ 6888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:07.188642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:07.188669 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:17:07.189112 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:07.189125 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.191351 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.191358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:07.193183 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:07.193228 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.196000 (+ 2772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.196009 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:07.196252 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:07.196284 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 20:17:07.196373 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 20:17:07.196506 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23669,"thread_start_us":23162,"threads_started":1}
I20260929 20:17:07.197544 3390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:07.158049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39524 (local address 127.27.185.66:33241)
0929 20:17:07.187983 (+ 29934us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:07.187997 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:07.188038 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:17:07.188150 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:07.188158 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:07.188421 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:07.188646 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:17:07.189750 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.191212 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.193349 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.194745 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.194870 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:07.195080 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:07.197103 (+ 2023us) server_negotiation.cc:299] Negotiation successful
0929 20:17:07.197280 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29698,"thread_start_us":29563,"threads_started":1}
I20260929 20:17:07.200207 3289 tablet_service.cc:1467] Processing CreateTablet for tablet 9c515e9015a249319a2208cc6a140813 (DEFAULT_TABLE table=test-table [id=c80ab07bfbc648e69e73b03d69cb3be8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:17:07.200207 3002 tablet_service.cc:1467] Processing CreateTablet for tablet 9c515e9015a249319a2208cc6a140813 (DEFAULT_TABLE table=test-table [id=c80ab07bfbc648e69e73b03d69cb3be8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:17:07.200425 3148 tablet_service.cc:1467] Processing CreateTablet for tablet 9c515e9015a249319a2208cc6a140813 (DEFAULT_TABLE table=test-table [id=c80ab07bfbc648e69e73b03d69cb3be8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:17:07.201606 3148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c515e9015a249319a2208cc6a140813. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:07.201678 3289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c515e9015a249319a2208cc6a140813. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:07.201704 3002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c515e9015a249319a2208cc6a140813. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:07.213670 3391 tablet_bootstrap.cc:492] T 9c515e9015a249319a2208cc6a140813 P 100246ec987b437eb04e30cf2815bced: Bootstrap starting.
I20260929 20:17:07.217195 3391 tablet_bootstrap.cc:654] T 9c515e9015a249319a2208cc6a140813 P 100246ec987b437eb04e30cf2815bced: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:07.217628 3392 tablet_bootstrap.cc:492] T 9c515e9015a249319a2208cc6a140813 P eaa3fccfe54b400e80a3f4c5c33a1baf: Bootstrap starting.
I20260929 20:17:07.218925 3391 log.cc:824] T 9c515e9015a249319a2208cc6a140813 P 100246ec987b437eb04e30cf2815bced: Log is configured to *not* fsync() on all Append() calls
I20260929 20:17:07.219750 3393 tablet_bootstrap.cc:492] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63: Bootstrap starting.
I20260929 20:17:07.221587 3392 tablet_bootstrap.cc:654] T 9c515e9015a249319a2208cc6a140813 P eaa3fccfe54b400e80a3f4c5c33a1baf: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:07.222904 3393 tablet_bootstrap.cc:654] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:07.223258 3392 log.cc:824] T 9c515e9015a249319a2208cc6a140813 P eaa3fccfe54b400e80a3f4c5c33a1baf: Log is configured to *not* fsync() on all Append() calls
I20260929 20:17:07.224282 3393 log.cc:824] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63: Log is configured to *not* fsync() on all Append() calls
I20260929 20:17:07.226917 3391 tablet_bootstrap.cc:492] T 9c515e9015a249319a2208cc6a140813 P 100246ec987b437eb04e30cf2815bced: No bootstrap required, opened a new log
I20260929 20:17:07.226917 3392 tablet_bootstrap.cc:492] T 9c515e9015a249319a2208cc6a140813 P eaa3fccfe54b400e80a3f4c5c33a1baf: No bootstrap required, opened a new log
I20260929 20:17:07.227217 3392 ts_tablet_manager.cc:1397] T 9c515e9015a249319a2208cc6a140813 P eaa3fccfe54b400e80a3f4c5c33a1baf: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260929 20:17:07.227217 3391 ts_tablet_manager.cc:1397] T 9c515e9015a249319a2208cc6a140813 P 100246ec987b437eb04e30cf2815bced: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260929 20:17:07.229225 3393 tablet_bootstrap.cc:492] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63: No bootstrap required, opened a new log
I20260929 20:17:07.229542 3393 ts_tablet_manager.cc:1397] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260929 20:17:07.237378 3393 raft_consensus.cc:348] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b9089a4cb642d682558e47961dcc63" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45841 } } peers { permanent_uuid: "eaa3fccfe54b400e80a3f4c5c33a1baf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42717 } } peers { permanent_uuid: "100246ec987b437eb04e30cf2815bced" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33241 } }
I20260929 20:17:07.237802 3393 raft_consensus.cc:374] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:07.237895 3393 raft_consensus.cc:729] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b9089a4cb642d682558e47961dcc63, State: Initialized, Role: FOLLOWER
I20260929 20:17:07.238351 3392 raft_consensus.cc:348] T 9c515e9015a249319a2208cc6a140813 P eaa3fccfe54b400e80a3f4c5c33a1baf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b9089a4cb642d682558e47961dcc63" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45841 } } peers { permanent_uuid: "eaa3fccfe54b400e80a3f4c5c33a1baf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42717 } } peers { permanent_uuid: "100246ec987b437eb04e30cf2815bced" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33241 } }
I20260929 20:17:07.238351 3391 raft_consensus.cc:348] T 9c515e9015a249319a2208cc6a140813 P 100246ec987b437eb04e30cf2815bced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b9089a4cb642d682558e47961dcc63" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45841 } } peers { permanent_uuid: "eaa3fccfe54b400e80a3f4c5c33a1baf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42717 } } peers { permanent_uuid: "100246ec987b437eb04e30cf2815bced" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33241 } }
I20260929 20:17:07.238571 3393 consensus_queue.cc:260] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b9089a4cb642d682558e47961dcc63" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45841 } } peers { permanent_uuid: "eaa3fccfe54b400e80a3f4c5c33a1baf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42717 } } peers { permanent_uuid: "100246ec987b437eb04e30cf2815bced" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33241 } }
I20260929 20:17:07.238801 3391 raft_consensus.cc:374] T 9c515e9015a249319a2208cc6a140813 P 100246ec987b437eb04e30cf2815bced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:07.238802 3392 raft_consensus.cc:374] T 9c515e9015a249319a2208cc6a140813 P eaa3fccfe54b400e80a3f4c5c33a1baf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:07.238899 3391 raft_consensus.cc:729] T 9c515e9015a249319a2208cc6a140813 P 100246ec987b437eb04e30cf2815bced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100246ec987b437eb04e30cf2815bced, State: Initialized, Role: FOLLOWER
I20260929 20:17:07.238899 3392 raft_consensus.cc:729] T 9c515e9015a249319a2208cc6a140813 P eaa3fccfe54b400e80a3f4c5c33a1baf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaa3fccfe54b400e80a3f4c5c33a1baf, State: Initialized, Role: FOLLOWER
I20260929 20:17:07.239740 3392 consensus_queue.cc:260] T 9c515e9015a249319a2208cc6a140813 P eaa3fccfe54b400e80a3f4c5c33a1baf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b9089a4cb642d682558e47961dcc63" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45841 } } peers { permanent_uuid: "eaa3fccfe54b400e80a3f4c5c33a1baf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42717 } } peers { permanent_uuid: "100246ec987b437eb04e30cf2815bced" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33241 } }
I20260929 20:17:07.240104 3355 heartbeater.cc:503] Master 127.27.185.126:36737 was elected leader, sending a full tablet report...
I20260929 20:17:07.240610 3393 ts_tablet_manager.cc:1428] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260929 20:17:07.241554 3391 consensus_queue.cc:260] T 9c515e9015a249319a2208cc6a140813 P 100246ec987b437eb04e30cf2815bced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b9089a4cb642d682558e47961dcc63" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45841 } } peers { permanent_uuid: "eaa3fccfe54b400e80a3f4c5c33a1baf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42717 } } peers { permanent_uuid: "100246ec987b437eb04e30cf2815bced" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33241 } }
I20260929 20:17:07.248121 3392 ts_tablet_manager.cc:1428] T 9c515e9015a249319a2208cc6a140813 P eaa3fccfe54b400e80a3f4c5c33a1baf: Time spent starting tablet: real 0.021s user 0.017s sys 0.005s
I20260929 20:17:07.252353 3391 ts_tablet_manager.cc:1428] T 9c515e9015a249319a2208cc6a140813 P 100246ec987b437eb04e30cf2815bced: Time spent starting tablet: real 0.025s user 0.014s sys 0.007s
I20260929 20:17:07.252859 3214 heartbeater.cc:503] Master 127.27.185.126:36737 was elected leader, sending a full tablet report...
W20260929 20:17:07.262517 3070 tablet.cc:2369] T 9c515e9015a249319a2208cc6a140813 P eaa3fccfe54b400e80a3f4c5c33a1baf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:17:07.282598 3397 raft_consensus.cc:482] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:17:07.282863 3397 raft_consensus.cc:504] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b9089a4cb642d682558e47961dcc63" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45841 } } peers { permanent_uuid: "eaa3fccfe54b400e80a3f4c5c33a1baf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42717 } } peers { permanent_uuid: "100246ec987b437eb04e30cf2815bced" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33241 } }
I20260929 20:17:07.284749 3397 leader_election.cc:288] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaa3fccfe54b400e80a3f4c5c33a1baf (127.27.185.65:42717), 100246ec987b437eb04e30cf2815bced (127.27.185.66:33241)
I20260929 20:17:07.290431 3390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:07.285216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:39471 (local address 127.27.185.66:33241)
0929 20:17:07.285464 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:07.285471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:07.285490 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:17:07.285564 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:07.285568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:07.285622 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:07.285725 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:17:07.286164 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.286874 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.288585 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.289386 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.289567 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:07.289643 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:07.290037 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 20:17:07.290173 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 20:17:07.290473 3388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:07.285453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:40769 (local address 127.27.185.65:42717)
0929 20:17:07.285647 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:07.285651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:07.285666 (+ 15us) server_negotiation.cc:407] Connection header received
0929 20:17:07.285732 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:07.285735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:07.285787 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:07.285894 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:17:07.286306 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.287004 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.288658 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.289438 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.289610 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:07.289684 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:07.290089 (+ 405us) server_negotiation.cc:299] Negotiation successful
0929 20:17:07.290224 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:17:07.290467 3363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:07.285002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:33241 (local address 127.27.185.67:39471)
0929 20:17:07.285280 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:07.285300 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:07.285410 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:07.285745 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:07.285749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:07.285773 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:17:07.286004 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:07.286013 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.287023 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.287030 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:07.288420 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:07.288432 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.289575 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.289581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:07.289840 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:07.289896 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 20:17:07.290005 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 20:17:07.290150 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":210}
I20260929 20:17:07.290474 3365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:07.284992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:42717 (local address 127.27.185.67:40769)
0929 20:17:07.285169 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:07.285191 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:07.285318 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:07.285919 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:07.285928 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:07.285952 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:17:07.286153 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:07.286160 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.287181 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.287184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:07.288508 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:07.288519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.289611 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.289617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:07.289917 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:07.289941 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 20:17:07.290051 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 20:17:07.290188 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":43}
I20260929 20:17:07.291529 3168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c515e9015a249319a2208cc6a140813" candidate_uuid: "67b9089a4cb642d682558e47961dcc63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100246ec987b437eb04e30cf2815bced" is_pre_election: true
I20260929 20:17:07.291916 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c515e9015a249319a2208cc6a140813" candidate_uuid: "67b9089a4cb642d682558e47961dcc63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaa3fccfe54b400e80a3f4c5c33a1baf" is_pre_election: true
I20260929 20:17:07.292161 3168 raft_consensus.cc:2454] T 9c515e9015a249319a2208cc6a140813 P 100246ec987b437eb04e30cf2815bced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b9089a4cb642d682558e47961dcc63 in term 0.
I20260929 20:17:07.292469 3022 raft_consensus.cc:2454] T 9c515e9015a249319a2208cc6a140813 P eaa3fccfe54b400e80a3f4c5c33a1baf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b9089a4cb642d682558e47961dcc63 in term 0.
I20260929 20:17:07.292920 3243 leader_election.cc:302] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100246ec987b437eb04e30cf2815bced, 67b9089a4cb642d682558e47961dcc63; no voters:
I20260929 20:17:07.293426 3397 raft_consensus.cc:2781] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:17:07.293602 3397 raft_consensus.cc:482] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:17:07.293725 3397 raft_consensus.cc:3037] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:17:07.296337 3397 raft_consensus.cc:504] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b9089a4cb642d682558e47961dcc63" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45841 } } peers { permanent_uuid: "eaa3fccfe54b400e80a3f4c5c33a1baf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42717 } } peers { permanent_uuid: "100246ec987b437eb04e30cf2815bced" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33241 } }
I20260929 20:17:07.296999 3397 leader_election.cc:288] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63 [CANDIDATE]: Term 1 election: Requested vote from peers eaa3fccfe54b400e80a3f4c5c33a1baf (127.27.185.65:42717), 100246ec987b437eb04e30cf2815bced (127.27.185.66:33241)
I20260929 20:17:07.297473 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c515e9015a249319a2208cc6a140813" candidate_uuid: "67b9089a4cb642d682558e47961dcc63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaa3fccfe54b400e80a3f4c5c33a1baf"
I20260929 20:17:07.297509 3168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c515e9015a249319a2208cc6a140813" candidate_uuid: "67b9089a4cb642d682558e47961dcc63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100246ec987b437eb04e30cf2815bced"
I20260929 20:17:07.297676 3022 raft_consensus.cc:3037] T 9c515e9015a249319a2208cc6a140813 P eaa3fccfe54b400e80a3f4c5c33a1baf [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:17:07.297731 3168 raft_consensus.cc:3037] T 9c515e9015a249319a2208cc6a140813 P 100246ec987b437eb04e30cf2815bced [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:17:07.300065 3022 raft_consensus.cc:2454] T 9c515e9015a249319a2208cc6a140813 P eaa3fccfe54b400e80a3f4c5c33a1baf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b9089a4cb642d682558e47961dcc63 in term 1.
I20260929 20:17:07.300135 3168 raft_consensus.cc:2454] T 9c515e9015a249319a2208cc6a140813 P 100246ec987b437eb04e30cf2815bced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b9089a4cb642d682558e47961dcc63 in term 1.
I20260929 20:17:07.300567 3245 leader_election.cc:302] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b9089a4cb642d682558e47961dcc63, eaa3fccfe54b400e80a3f4c5c33a1baf; no voters:
I20260929 20:17:07.300985 3397 raft_consensus.cc:2781] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:17:07.301436 3397 raft_consensus.cc:686] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63 [term 1 LEADER]: Becoming Leader. State: Replica: 67b9089a4cb642d682558e47961dcc63, State: Running, Role: LEADER
I20260929 20:17:07.302038 3397 consensus_queue.cc:237] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b9089a4cb642d682558e47961dcc63" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45841 } } peers { permanent_uuid: "eaa3fccfe54b400e80a3f4c5c33a1baf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42717 } } peers { permanent_uuid: "100246ec987b437eb04e30cf2815bced" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33241 } }
I20260929 20:17:07.307191 2100 catalog_manager.cc:5495] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67b9089a4cb642d682558e47961dcc63 (127.27.185.67). New cstate: current_term: 1 leader_uuid: "67b9089a4cb642d682558e47961dcc63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b9089a4cb642d682558e47961dcc63" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaa3fccfe54b400e80a3f4c5c33a1baf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "100246ec987b437eb04e30cf2815bced" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33241 } health_report { overall_health: UNKNOWN } } }
W20260929 20:17:07.333704 3360 tablet.cc:2369] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:17:07.362025 3386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:07.353835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41846 (local address 127.27.185.67:45841)
0929 20:17:07.354102 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:07.354115 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:07.354140 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:17:07.354274 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:07.354290 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:07.354394 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:07.354606 (+ 212us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:17:07.354941 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.356699 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.358212 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.359210 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.359485 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:07.360077 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:17:07.361174 (+ 1097us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:17:07.361447 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:07.361523 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 20:17:07.361647 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 20:17:07.371069 3168 raft_consensus.cc:1261] T 9c515e9015a249319a2208cc6a140813 P 100246ec987b437eb04e30cf2815bced [term 1 FOLLOWER]: Refusing update from remote peer 67b9089a4cb642d682558e47961dcc63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:17:07.371070 3022 raft_consensus.cc:1261] T 9c515e9015a249319a2208cc6a140813 P eaa3fccfe54b400e80a3f4c5c33a1baf [term 1 FOLLOWER]: Refusing update from remote peer 67b9089a4cb642d682558e47961dcc63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:17:07.372242 3397 consensus_queue.cc:1035] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100246ec987b437eb04e30cf2815bced" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:17:07.372561 3400 consensus_queue.cc:1035] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaa3fccfe54b400e80a3f4c5c33a1baf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:17:07.417568 2101 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:17:07.417894 2101 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:17:07.425536 2101 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a60def670884f8886ada0b1616e4159: Sending DeleteTablet for 3 replicas of tablet 9c515e9015a249319a2208cc6a140813
I20260929 20:17:07.426618 3289 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c515e9015a249319a2208cc6a140813 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:17:07 UTC) from {username='oryx', principal='oryx/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:41844
I20260929 20:17:07.426725 3002 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c515e9015a249319a2208cc6a140813 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:17:07 UTC) from {username='oryx', principal='oryx/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:39480
I20260929 20:17:07.427476 3148 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c515e9015a249319a2208cc6a140813 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:17:07 UTC) from {username='oryx', principal='oryx/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:39524
I20260929 20:17:07.440023 3415 tablet_replica.cc:330] T 9c515e9015a249319a2208cc6a140813 P 100246ec987b437eb04e30cf2815bced: stopping tablet replica
I20260929 20:17:07.440629 3415 raft_consensus.cc:2229] T 9c515e9015a249319a2208cc6a140813 P 100246ec987b437eb04e30cf2815bced [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:17:07.441288 3415 raft_consensus.cc:2258] T 9c515e9015a249319a2208cc6a140813 P 100246ec987b437eb04e30cf2815bced [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:17:07.443563 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 2069
I20260929 20:17:07.444934 2131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:07.445341 3415 ts_tablet_manager.cc:1905] T 9c515e9015a249319a2208cc6a140813 P 100246ec987b437eb04e30cf2815bced: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:17:07.445502 3414 tablet_replica.cc:330] T 9c515e9015a249319a2208cc6a140813 P eaa3fccfe54b400e80a3f4c5c33a1baf: stopping tablet replica
I20260929 20:17:07.445941 3414 raft_consensus.cc:2229] T 9c515e9015a249319a2208cc6a140813 P eaa3fccfe54b400e80a3f4c5c33a1baf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:17:07.446506 3414 raft_consensus.cc:2258] T 9c515e9015a249319a2208cc6a140813 P eaa3fccfe54b400e80a3f4c5c33a1baf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:17:07.450245 3414 ts_tablet_manager.cc:1905] T 9c515e9015a249319a2208cc6a140813 P eaa3fccfe54b400e80a3f4c5c33a1baf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:17:07.452939 3413 tablet_replica.cc:330] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63: stopping tablet replica
I20260929 20:17:07.453377 3413 raft_consensus.cc:2229] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:17:07.453733 3415 ts_tablet_manager.cc:1918] T 9c515e9015a249319a2208cc6a140813 P 100246ec987b437eb04e30cf2815bced: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:17:07.454811 3413 raft_consensus.cc:2258] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:17:07.457629 3415 log.cc:1196] T 9c515e9015a249319a2208cc6a140813 P 100246ec987b437eb04e30cf2815bced: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/wal/wals/9c515e9015a249319a2208cc6a140813
I20260929 20:17:07.458253 3415 ts_tablet_manager.cc:1939] T 9c515e9015a249319a2208cc6a140813 P 100246ec987b437eb04e30cf2815bced: Deleting consensus metadata
I20260929 20:17:07.458530 3413 ts_tablet_manager.cc:1905] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:17:07.470638 3413 ts_tablet_manager.cc:1918] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:17:07.470819 3413 log.cc:1196] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/wal/wals/9c515e9015a249319a2208cc6a140813
I20260929 20:17:07.471447 3413 ts_tablet_manager.cc:1939] T 9c515e9015a249319a2208cc6a140813 P 67b9089a4cb642d682558e47961dcc63: Deleting consensus metadata
I20260929 20:17:07.473690 3414 ts_tablet_manager.cc:1918] T 9c515e9015a249319a2208cc6a140813 P eaa3fccfe54b400e80a3f4c5c33a1baf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:17:07.473863 3414 log.cc:1196] T 9c515e9015a249319a2208cc6a140813 P eaa3fccfe54b400e80a3f4c5c33a1baf: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/wal/wals/9c515e9015a249319a2208cc6a140813
I20260929 20:17:07.474464 3414 ts_tablet_manager.cc:1939] T 9c515e9015a249319a2208cc6a140813 P eaa3fccfe54b400e80a3f4c5c33a1baf: Deleting consensus metadata
I20260929 20:17:07.476186 3370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:07.429806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34318 (local address 127.27.185.125:46841)
0929 20:17:07.430049 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:07.430058 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:07.442664 (+ 12606us) server_negotiation.cc:407] Connection header received
0929 20:17:07.451850 (+ 9186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:07.451861 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:07.451994 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:07.452134 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:07.456227 (+ 4093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.457333 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.462942 (+ 5609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.463661 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.463817 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:07.467915 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:07.467922 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:07.467928 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:07.467971 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:07.471767 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:07.472866 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:07.472872 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:07.472875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:07.473036 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:07.473553 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:07.473558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:07.473561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:07.473963 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:07.474212 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:07.475598 (+ 1386us) server_negotiation.cc:299] Negotiation successful
0929 20:17:07.475824 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 20:17:07.489125 3358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:07.442545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55040 (local address 127.27.185.124:42155)
0929 20:17:07.442856 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:07.442864 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:07.475367 (+ 32503us) server_negotiation.cc:407] Connection header received
0929 20:17:07.475626 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:07.475631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:07.475717 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:07.475793 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:07.477786 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.478755 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.480089 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.480514 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.480652 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:07.483156 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:07.483163 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:07.483168 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:07.483210 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:07.486339 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:07.486977 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:07.486983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:07.486987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:07.487148 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:07.487565 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:07.487572 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:07.487575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:07.487863 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:07.488195 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:07.488492 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 20:17:07.488706 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260929 20:17:07.649221 2086 catalog_manager.cc:4841] TS 100246ec987b437eb04e30cf2815bced (127.27.185.66:33241): tablet 9c515e9015a249319a2208cc6a140813 (table test-table [id=c80ab07bfbc648e69e73b03d69cb3be8]) successfully deleted
I20260929 20:17:07.649359 2085 catalog_manager.cc:4841] TS 67b9089a4cb642d682558e47961dcc63 (127.27.185.67:45841): tablet 9c515e9015a249319a2208cc6a140813 (table test-table [id=c80ab07bfbc648e69e73b03d69cb3be8]) successfully deleted
I20260929 20:17:07.649524 2088 catalog_manager.cc:4841] TS eaa3fccfe54b400e80a3f4c5c33a1baf (127.27.185.65:42717): tablet 9c515e9015a249319a2208cc6a140813 (table test-table [id=c80ab07bfbc648e69e73b03d69cb3be8]) successfully deleted
I20260929 20:17:07.649996 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 2069
I20260929 20:17:07.695093 3358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:07.677596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55050 (local address 127.27.185.124:42155)
0929 20:17:07.677812 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:07.677818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:07.679425 (+ 1607us) server_negotiation.cc:407] Connection header received
0929 20:17:07.679668 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:07.679673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:07.679748 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:07.679854 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:07.681508 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.682658 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.684693 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.685323 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.685475 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:07.688340 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:07.688345 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:07.688348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:07.688384 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:07.691880 (+ 3496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:07.692786 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:07.692791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:07.692798 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:07.692951 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:07.693379 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:07.693384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:07.693388 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:07.693777 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:07.694029 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:07.694448 (+ 419us) server_negotiation.cc:299] Negotiation successful
0929 20:17:07.694681 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:17:07.695996 3370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:07.677518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34320 (local address 127.27.185.125:46841)
0929 20:17:07.677771 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:07.677780 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:07.678804 (+ 1024us) server_negotiation.cc:407] Connection header received
0929 20:17:07.679116 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:07.679123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:07.679211 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:07.679285 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:07.681555 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.682658 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.684244 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:07.684741 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:07.684871 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:07.688974 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:07.688989 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:07.688997 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:07.689062 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:07.692718 (+ 3656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:07.693446 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:07.693452 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:07.693456 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:07.693588 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:07.694018 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:07.694023 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:07.694027 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:07.694402 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:07.694636 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:07.695434 (+ 798us) server_negotiation.cc:299] Negotiation successful
0929 20:17:07.695651 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 20:17:08.401191 3425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:08.400254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.27.185.67:33621)
0929 20:17:08.400744 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:08.400935 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":247,"threads_started":1}
W20260929 20:17:08.401844 3355 heartbeater.cc:644] Failed to heartbeat to 127.27.185.126:36737 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
I20260929 20:17:08.404680 3426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:08.403925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.27.185.66:56135)
0929 20:17:08.404425 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:08.404539 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":256,"threads_started":1}
W20260929 20:17:08.405287 3214 heartbeater.cc:644] Failed to heartbeat to 127.27.185.126:36737 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
I20260929 20:17:08.409184 3427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:08.408337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.27.185.65:41299)
0929 20:17:08.408804 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:08.408952 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":225,"threads_started":1}
W20260929 20:17:08.409775 3068 heartbeater.cc:644] Failed to heartbeat to 127.27.185.126:36737 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
I20260929 20:17:08.925038 3428 raft_consensus.cc:482] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a60def670884f8886ada0b1616e4159)
I20260929 20:17:08.925319 3428 raft_consensus.cc:504] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: false } }
I20260929 20:17:08.926295 3429 raft_consensus.cc:482] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a60def670884f8886ada0b1616e4159)
I20260929 20:17:08.926642 3428 leader_election.cc:288] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a60def670884f8886ada0b1616e4159 (127.27.185.126:36737), ceb7a8c34c984b92ab34d3a2e1d14d49 (127.27.185.125:46841)
I20260929 20:17:08.926532 3429 raft_consensus.cc:504] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: false } }
I20260929 20:17:08.938227 3431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:08.926728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.0.0.1:34932)
0929 20:17:08.937537 (+ 10809us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:08.937878 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10601,"threads_started":1}
I20260929 20:17:08.939405 3433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:08.928455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.0.0.1:34938)
0929 20:17:08.938867 (+ 10412us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:08.939052 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10298,"thread_start_us":10172,"threads_started":1}
W20260929 20:17:08.939679 2688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
I20260929 20:17:08.941743 3429 leader_election.cc:288] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a60def670884f8886ada0b1616e4159 (127.27.185.126:36737), 927955a2d44a4c7198c39f371d6671d3 (127.27.185.124:42155)
W20260929 20:17:08.945946 2876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
I20260929 20:17:08.946192 3434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:08.944782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.0.0.1:34950)
0929 20:17:08.945701 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:08.945943 (+ 242us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":787}
W20260929 20:17:08.946710 2688 leader_election.cc:334] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3a60def670884f8886ada0b1616e4159 (127.27.185.126:36737): Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
I20260929 20:17:08.947187 3430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:08.926817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:46841 (local address 127.0.0.1:34322)
0929 20:17:08.937898 (+ 11081us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:08.937922 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:08.938105 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:08.939450 (+ 1345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:08.939456 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:08.939606 (+ 150us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:17:08.940001 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:08.940017 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:08.942233 (+ 2216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:08.942241 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:08.944762 (+ 2521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:08.944778 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:08.946262 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:08.946267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:08.946505 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:08.946550 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 20:17:08.946645 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 20:17:08.946778 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10911,"thread_start_us":10350,"threads_started":1}
I20260929 20:17:08.948342 3432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:08.927016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34322 (local address 127.27.185.125:46841)
0929 20:17:08.938867 (+ 11851us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:08.938874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:08.938894 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:17:08.938963 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:08.938973 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:08.939062 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:08.939163 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:17:08.940221 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:08.942027 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:08.944937 (+ 2910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:08.946042 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:08.947701 (+ 1659us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:08.947777 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:08.947825 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 20:17:08.947946 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11735,"thread_start_us":13033,"threads_started":1}
I20260929 20:17:08.948680 3430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:08.948275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.0.0.1:34966)
0929 20:17:08.948451 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:08.948562 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 20:17:08.948938 2876 leader_election.cc:334] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3a60def670884f8886ada0b1616e4159 (127.27.185.126:36737): Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
I20260929 20:17:08.949249 2711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "927955a2d44a4c7198c39f371d6671d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" is_pre_election: true
I20260929 20:17:08.949602 2711 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927955a2d44a4c7198c39f371d6671d3 in term 1.
I20260929 20:17:08.950078 2873 leader_election.cc:302] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 927955a2d44a4c7198c39f371d6671d3, ceb7a8c34c984b92ab34d3a2e1d14d49; no voters: 3a60def670884f8886ada0b1616e4159
I20260929 20:17:08.950456 3428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 20:17:08.950588 3428 raft_consensus.cc:482] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3a60def670884f8886ada0b1616e4159)
I20260929 20:17:08.950730 3428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [term 1 FOLLOWER]: Advancing to term 2
I20260929 20:17:08.952584 3433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:08.934821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:42155 (local address 127.0.0.1:55056)
0929 20:17:08.943103 (+ 8282us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:08.943121 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:08.943229 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:08.945592 (+ 2363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:08.945597 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:08.945630 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:17:08.945965 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:08.945978 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:08.948228 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:08.948235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:08.950334 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:08.950347 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:08.951819 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:08.951825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:08.952021 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:08.952045 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 20:17:08.952154 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 20:17:08.952292 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8185,"thread_start_us":11474,"threads_started":1}
I20260929 20:17:08.952677 3435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:08.938686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55056 (local address 127.27.185.124:42155)
0929 20:17:08.945226 (+ 6540us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:08.945235 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:08.945259 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:17:08.945326 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:08.945332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:08.945412 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:08.945543 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:17:08.946224 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:08.948080 (+ 1856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:08.950515 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:08.951619 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:08.951824 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:08.951904 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:08.952179 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 20:17:08.952313 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6384,"thread_start_us":7050,"threads_started":1}
I20260929 20:17:08.953225 2899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "927955a2d44a4c7198c39f371d6671d3" is_pre_election: true
I20260929 20:17:08.953433 3428 raft_consensus.cc:504] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: false } }
I20260929 20:17:08.953873 2899 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ceb7a8c34c984b92ab34d3a2e1d14d49 in current term 2: Already voted for candidate 927955a2d44a4c7198c39f371d6671d3 in this term.
I20260929 20:17:08.954160 3428 leader_election.cc:288] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [CANDIDATE]: Term 2 election: Requested vote from peers 3a60def670884f8886ada0b1616e4159 (127.27.185.126:36737), ceb7a8c34c984b92ab34d3a2e1d14d49 (127.27.185.125:46841)
I20260929 20:17:08.954593 2711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "927955a2d44a4c7198c39f371d6671d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49"
I20260929 20:17:08.954665 3430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:08.954254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.0.0.1:34974)
0929 20:17:08.954431 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:08.954527 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 20:17:08.954783 2686 leader_election.cc:302] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ceb7a8c34c984b92ab34d3a2e1d14d49; no voters: 3a60def670884f8886ada0b1616e4159, 927955a2d44a4c7198c39f371d6671d3
I20260929 20:17:08.954833 2711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [term 1 FOLLOWER]: Advancing to term 2
I20260929 20:17:08.955775 3430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:08.955411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.0.0.1:34984)
0929 20:17:08.955545 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:08.955647 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 20:17:08.955960 2876 leader_election.cc:334] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3a60def670884f8886ada0b1616e4159 (127.27.185.126:36737): Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
I20260929 20:17:08.957538 2711 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927955a2d44a4c7198c39f371d6671d3 in term 2.
I20260929 20:17:08.957715 3429 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 20:17:08.957930 2873 leader_election.cc:302] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 927955a2d44a4c7198c39f371d6671d3, ceb7a8c34c984b92ab34d3a2e1d14d49; no voters: 3a60def670884f8886ada0b1616e4159
I20260929 20:17:08.958214 3428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 20:17:08.958638 3428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [term 2 LEADER]: Becoming Leader. State: Replica: 927955a2d44a4c7198c39f371d6671d3, State: Running, Role: LEADER
I20260929 20:17:08.959028 3428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: false } }
I20260929 20:17:08.960871 3438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 927955a2d44a4c7198c39f371d6671d3. Latest consensus state: current_term: 2 leader_uuid: "927955a2d44a4c7198c39f371d6671d3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: false } } }
I20260929 20:17:08.961175 3438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [sys.catalog]: This master's current role is: LEADER
I20260929 20:17:08.961833 3440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:17:08.965549 3440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:17:08.966360 3440 catalog_manager.cc:1259] Loaded cluster ID: 09b63d71bcc342fcaf7a45474598022f
I20260929 20:17:08.966470 3440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:17:08.967653 3440 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:17:08.968441 3440 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3: Loaded TSK: 0
I20260929 20:17:08.969312 3440 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:17:08.979605 2889 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:17:08.986917 2889 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:17:08.991044 3430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:08.990596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.0.0.1:34998)
0929 20:17:08.990788 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:08.990891 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 20:17:08.991240 2711 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [term 2 FOLLOWER]: Refusing update from remote peer 927955a2d44a4c7198c39f371d6671d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 20:17:08.992202 3430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:08.991832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.0.0.1:35008)
0929 20:17:08.991985 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:08.992065 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 20:17:08.992030 3438 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260929 20:17:08.992452 2876 consensus_peers.cc:487] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 -> Peer 3a60def670884f8886ada0b1616e4159 (127.27.185.126:36737): Couldn't send request to peer 3a60def670884f8886ada0b1616e4159. Status: Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 20:17:08.995013 3429 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 927955a2d44a4c7198c39f371d6671d3. Latest consensus state: current_term: 2 leader_uuid: "927955a2d44a4c7198c39f371d6671d3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: false } } }
I20260929 20:17:08.995287 3429 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:17:08.996723 3428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "927955a2d44a4c7198c39f371d6671d3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: false } } }
I20260929 20:17:08.997076 3428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3 [sys.catalog]: This master's current role is: LEADER
I20260929 20:17:08.997278 3429 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "927955a2d44a4c7198c39f371d6671d3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3a60def670884f8886ada0b1616e4159" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36737 } } peers { permanent_uuid: "ceb7a8c34c984b92ab34d3a2e1d14d49" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "927955a2d44a4c7198c39f371d6671d3" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 42155 } attrs { promote: false } } }
I20260929 20:17:08.997594 3429 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb7a8c34c984b92ab34d3a2e1d14d49 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:17:09.014732 3431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:09.005481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:45841 (local address 127.0.0.1:41862)
0929 20:17:09.005628 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:09.005647 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:09.005748 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:09.006713 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:09.006718 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:09.006750 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:17:09.007195 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:09.007210 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:09.009014 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:09.009022 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:09.011578 (+ 2556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:09.011590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:09.013692 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:09.013702 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:09.013980 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:09.014041 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 20:17:09.014126 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 20:17:09.014313 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 20:17:09.016171 3448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:09.005844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41862 (local address 127.27.185.67:45841)
0929 20:17:09.006331 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:09.006335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:09.006350 (+ 15us) server_negotiation.cc:407] Connection header received
0929 20:17:09.006476 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:09.006481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:09.006559 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:09.007376 (+ 817us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:17:09.007610 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:09.008813 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:09.011780 (+ 2967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:09.013382 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:09.015451 (+ 2069us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:09.015589 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:09.015642 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 20:17:09.015781 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":264,"threads_started":1}
I20260929 20:17:09.016975 3289 tablet_service.cc:1467] Processing CreateTablet for tablet cb82a02976de4814ac2a73ac5db0aa20 (DEFAULT_TABLE table=test-table [id=272720b5181d4d71b72ee4056fd5922b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:17:09.017486 3289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb82a02976de4814ac2a73ac5db0aa20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:09.023505 3430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:09.005066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:33241 (local address 127.0.0.1:39526)
0929 20:17:09.005456 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:09.005475 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:09.005615 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:09.016687 (+ 11072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:09.016695 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:09.016733 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:17:09.016973 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:09.016987 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:09.018320 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:09.018329 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:09.020773 (+ 2444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:09.020786 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:09.022416 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:09.022425 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:09.022680 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:09.022736 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 20:17:09.022855 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 20:17:09.023065 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277}
I20260929 20:17:09.023841 3447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:09.005182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39526 (local address 127.27.185.66:33241)
0929 20:17:09.016316 (+ 11134us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:09.016324 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:09.016345 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:17:09.016422 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:09.016429 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:09.016520 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:09.016632 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:17:09.017159 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:09.018157 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:09.020925 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:09.022232 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:09.022355 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:09.022449 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:09.022921 (+ 472us) server_negotiation.cc:299] Negotiation successful
0929 20:17:09.023082 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10995,"thread_start_us":11557,"threads_started":1}
I20260929 20:17:09.024827 3148 tablet_service.cc:1467] Processing CreateTablet for tablet cb82a02976de4814ac2a73ac5db0aa20 (DEFAULT_TABLE table=test-table [id=272720b5181d4d71b72ee4056fd5922b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:17:09.025417 3148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb82a02976de4814ac2a73ac5db0aa20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:09.032433 3451 tablet_bootstrap.cc:492] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63: Bootstrap starting.
I20260929 20:17:09.033990 3431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:09.006479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:42717 (local address 127.0.0.1:39486)
0929 20:17:09.015017 (+ 8538us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:09.015036 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:09.015142 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:09.026373 (+ 11231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:09.026379 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:09.026407 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:17:09.026557 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:09.026566 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:09.028518 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:09.028527 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:09.031049 (+ 2522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:09.031072 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:09.033079 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:09.033088 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:09.033281 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:09.033331 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 20:17:09.033414 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 20:17:09.033629 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8424,"thread_start_us":9521,"threads_started":1}
I20260929 20:17:09.035493 3450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:09.006679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39486 (local address 127.27.185.65:42717)
0929 20:17:09.025908 (+ 19229us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:09.025917 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:09.025938 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:17:09.026013 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:09.026018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:09.026095 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:09.026710 (+ 615us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:17:09.027227 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:09.028330 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:09.031318 (+ 2988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:09.032868 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:09.034826 (+ 1958us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:09.034940 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:09.034997 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 20:17:09.035141 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19088,"thread_start_us":19509,"threads_started":1}
I20260929 20:17:09.036204 3451 tablet_bootstrap.cc:654] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:09.038396 3451 tablet_bootstrap.cc:492] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63: No bootstrap required, opened a new log
I20260929 20:17:09.038573 3451 ts_tablet_manager.cc:1397] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 20:17:09.039309 3451 raft_consensus.cc:348] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100246ec987b437eb04e30cf2815bced" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33241 } } peers { permanent_uuid: "67b9089a4cb642d682558e47961dcc63" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45841 } } peers { permanent_uuid: "eaa3fccfe54b400e80a3f4c5c33a1baf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42717 } }
I20260929 20:17:09.039592 3451 raft_consensus.cc:374] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:09.039692 3451 raft_consensus.cc:729] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b9089a4cb642d682558e47961dcc63, State: Initialized, Role: FOLLOWER
I20260929 20:17:09.039978 3451 consensus_queue.cc:260] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100246ec987b437eb04e30cf2815bced" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33241 } } peers { permanent_uuid: "67b9089a4cb642d682558e47961dcc63" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45841 } } peers { permanent_uuid: "eaa3fccfe54b400e80a3f4c5c33a1baf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42717 } }
I20260929 20:17:09.040876 3451 ts_tablet_manager.cc:1428] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 20:17:09.042778 3455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:09.041639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.27.185.67:33453)
0929 20:17:09.042198 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:09.042469 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"threads_started":1}
I20260929 20:17:09.045969 3455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:09.045091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.27.185.67:38327)
0929 20:17:09.045713 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:09.045801 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516}
I20260929 20:17:09.046373 3354 heartbeater.cc:503] Master 127.27.185.124:42155 was elected leader, sending a full tablet report...
I20260929 20:17:09.046761 3002 tablet_service.cc:1467] Processing CreateTablet for tablet cb82a02976de4814ac2a73ac5db0aa20 (DEFAULT_TABLE table=test-table [id=272720b5181d4d71b72ee4056fd5922b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:17:09.047271 3002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb82a02976de4814ac2a73ac5db0aa20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:09.049222 3452 tablet_bootstrap.cc:492] T cb82a02976de4814ac2a73ac5db0aa20 P 100246ec987b437eb04e30cf2815bced: Bootstrap starting.
I20260929 20:17:09.050772 3452 tablet_bootstrap.cc:654] T cb82a02976de4814ac2a73ac5db0aa20 P 100246ec987b437eb04e30cf2815bced: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:09.052815 3457 tablet_bootstrap.cc:492] T cb82a02976de4814ac2a73ac5db0aa20 P eaa3fccfe54b400e80a3f4c5c33a1baf: Bootstrap starting.
I20260929 20:17:09.052955 3452 tablet_bootstrap.cc:492] T cb82a02976de4814ac2a73ac5db0aa20 P 100246ec987b437eb04e30cf2815bced: No bootstrap required, opened a new log
I20260929 20:17:09.053108 3452 ts_tablet_manager.cc:1397] T cb82a02976de4814ac2a73ac5db0aa20 P 100246ec987b437eb04e30cf2815bced: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 20:17:09.053964 3452 raft_consensus.cc:348] T cb82a02976de4814ac2a73ac5db0aa20 P 100246ec987b437eb04e30cf2815bced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100246ec987b437eb04e30cf2815bced" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33241 } } peers { permanent_uuid: "67b9089a4cb642d682558e47961dcc63" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45841 } } peers { permanent_uuid: "eaa3fccfe54b400e80a3f4c5c33a1baf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42717 } }
I20260929 20:17:09.054232 3452 raft_consensus.cc:374] T cb82a02976de4814ac2a73ac5db0aa20 P 100246ec987b437eb04e30cf2815bced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:09.054330 3452 raft_consensus.cc:729] T cb82a02976de4814ac2a73ac5db0aa20 P 100246ec987b437eb04e30cf2815bced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100246ec987b437eb04e30cf2815bced, State: Initialized, Role: FOLLOWER
I20260929 20:17:09.054695 3457 tablet_bootstrap.cc:654] T cb82a02976de4814ac2a73ac5db0aa20 P eaa3fccfe54b400e80a3f4c5c33a1baf: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:09.054598 3452 consensus_queue.cc:260] T cb82a02976de4814ac2a73ac5db0aa20 P 100246ec987b437eb04e30cf2815bced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100246ec987b437eb04e30cf2815bced" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33241 } } peers { permanent_uuid: "67b9089a4cb642d682558e47961dcc63" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45841 } } peers { permanent_uuid: "eaa3fccfe54b400e80a3f4c5c33a1baf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42717 } }
I20260929 20:17:09.055500 3452 ts_tablet_manager.cc:1428] T cb82a02976de4814ac2a73ac5db0aa20 P 100246ec987b437eb04e30cf2815bced: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 20:17:09.057920 3213 heartbeater.cc:503] Master 127.27.185.124:42155 was elected leader, sending a full tablet report...
I20260929 20:17:09.058434 3460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:09.056102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.27.185.66:57301)
0929 20:17:09.057947 (+ 1845us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:09.058139 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1698,"thread_start_us":1556,"threads_started":1}
I20260929 20:17:09.059489 3460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:09.059003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.27.185.66:47613)
0929 20:17:09.059219 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:09.059304 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260929 20:17:09.060024 3457 tablet_bootstrap.cc:492] T cb82a02976de4814ac2a73ac5db0aa20 P eaa3fccfe54b400e80a3f4c5c33a1baf: No bootstrap required, opened a new log
I20260929 20:17:09.060140 3457 ts_tablet_manager.cc:1397] T cb82a02976de4814ac2a73ac5db0aa20 P eaa3fccfe54b400e80a3f4c5c33a1baf: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 20:17:09.060828 3457 raft_consensus.cc:348] T cb82a02976de4814ac2a73ac5db0aa20 P eaa3fccfe54b400e80a3f4c5c33a1baf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100246ec987b437eb04e30cf2815bced" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33241 } } peers { permanent_uuid: "67b9089a4cb642d682558e47961dcc63" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45841 } } peers { permanent_uuid: "eaa3fccfe54b400e80a3f4c5c33a1baf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42717 } }
I20260929 20:17:09.061024 3457 raft_consensus.cc:374] T cb82a02976de4814ac2a73ac5db0aa20 P eaa3fccfe54b400e80a3f4c5c33a1baf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:09.061092 3457 raft_consensus.cc:729] T cb82a02976de4814ac2a73ac5db0aa20 P eaa3fccfe54b400e80a3f4c5c33a1baf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaa3fccfe54b400e80a3f4c5c33a1baf, State: Initialized, Role: FOLLOWER
I20260929 20:17:09.061275 3457 consensus_queue.cc:260] T cb82a02976de4814ac2a73ac5db0aa20 P eaa3fccfe54b400e80a3f4c5c33a1baf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100246ec987b437eb04e30cf2815bced" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33241 } } peers { permanent_uuid: "67b9089a4cb642d682558e47961dcc63" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45841 } } peers { permanent_uuid: "eaa3fccfe54b400e80a3f4c5c33a1baf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42717 } }
I20260929 20:17:09.062311 3457 ts_tablet_manager.cc:1428] T cb82a02976de4814ac2a73ac5db0aa20 P eaa3fccfe54b400e80a3f4c5c33a1baf: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 20:17:09.063656 3462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:09.062620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.27.185.65:56141)
0929 20:17:09.063209 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:09.063377 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":281,"threads_started":1}
I20260929 20:17:09.065034 3067 heartbeater.cc:503] Master 127.27.185.124:42155 was elected leader, sending a full tablet report...
I20260929 20:17:09.065887 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:09.064274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.27.185.65:36981)
0929 20:17:09.064808 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:09.065594 (+ 786us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":260,"threads_started":1}
W20260929 20:17:09.159235 3216 tablet.cc:2369] T cb82a02976de4814ac2a73ac5db0aa20 P 100246ec987b437eb04e30cf2815bced: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:17:09.277972 3454 raft_consensus.cc:482] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:17:09.278226 3454 raft_consensus.cc:504] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100246ec987b437eb04e30cf2815bced" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33241 } } peers { permanent_uuid: "67b9089a4cb642d682558e47961dcc63" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45841 } } peers { permanent_uuid: "eaa3fccfe54b400e80a3f4c5c33a1baf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42717 } }
I20260929 20:17:09.279081 3454 leader_election.cc:288] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 100246ec987b437eb04e30cf2815bced (127.27.185.66:33241), eaa3fccfe54b400e80a3f4c5c33a1baf (127.27.185.65:42717)
I20260929 20:17:09.279762 3168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb82a02976de4814ac2a73ac5db0aa20" candidate_uuid: "67b9089a4cb642d682558e47961dcc63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100246ec987b437eb04e30cf2815bced" is_pre_election: true
I20260929 20:17:09.279762 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb82a02976de4814ac2a73ac5db0aa20" candidate_uuid: "67b9089a4cb642d682558e47961dcc63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaa3fccfe54b400e80a3f4c5c33a1baf" is_pre_election: true
I20260929 20:17:09.280043 3168 raft_consensus.cc:2454] T cb82a02976de4814ac2a73ac5db0aa20 P 100246ec987b437eb04e30cf2815bced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b9089a4cb642d682558e47961dcc63 in term 0.
I20260929 20:17:09.280056 3022 raft_consensus.cc:2454] T cb82a02976de4814ac2a73ac5db0aa20 P eaa3fccfe54b400e80a3f4c5c33a1baf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b9089a4cb642d682558e47961dcc63 in term 0.
I20260929 20:17:09.280546 3245 leader_election.cc:302] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 100246ec987b437eb04e30cf2815bced, 67b9089a4cb642d682558e47961dcc63, eaa3fccfe54b400e80a3f4c5c33a1baf; no voters:
I20260929 20:17:09.280869 3454 raft_consensus.cc:2781] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:17:09.281013 3454 raft_consensus.cc:482] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:17:09.281121 3454 raft_consensus.cc:3037] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:17:09.283254 3454 raft_consensus.cc:504] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100246ec987b437eb04e30cf2815bced" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33241 } } peers { permanent_uuid: "67b9089a4cb642d682558e47961dcc63" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45841 } } peers { permanent_uuid: "eaa3fccfe54b400e80a3f4c5c33a1baf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42717 } }
I20260929 20:17:09.283845 3454 leader_election.cc:288] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63 [CANDIDATE]: Term 1 election: Requested vote from peers 100246ec987b437eb04e30cf2815bced (127.27.185.66:33241), eaa3fccfe54b400e80a3f4c5c33a1baf (127.27.185.65:42717)
I20260929 20:17:09.284394 3168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb82a02976de4814ac2a73ac5db0aa20" candidate_uuid: "67b9089a4cb642d682558e47961dcc63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100246ec987b437eb04e30cf2815bced"
I20260929 20:17:09.284425 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb82a02976de4814ac2a73ac5db0aa20" candidate_uuid: "67b9089a4cb642d682558e47961dcc63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaa3fccfe54b400e80a3f4c5c33a1baf"
I20260929 20:17:09.284606 3168 raft_consensus.cc:3037] T cb82a02976de4814ac2a73ac5db0aa20 P 100246ec987b437eb04e30cf2815bced [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:17:09.284642 3022 raft_consensus.cc:3037] T cb82a02976de4814ac2a73ac5db0aa20 P eaa3fccfe54b400e80a3f4c5c33a1baf [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:17:09.286721 3168 raft_consensus.cc:2454] T cb82a02976de4814ac2a73ac5db0aa20 P 100246ec987b437eb04e30cf2815bced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b9089a4cb642d682558e47961dcc63 in term 1.
I20260929 20:17:09.286828 3022 raft_consensus.cc:2454] T cb82a02976de4814ac2a73ac5db0aa20 P eaa3fccfe54b400e80a3f4c5c33a1baf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b9089a4cb642d682558e47961dcc63 in term 1.
I20260929 20:17:09.287197 3243 leader_election.cc:302] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100246ec987b437eb04e30cf2815bced, 67b9089a4cb642d682558e47961dcc63; no voters:
I20260929 20:17:09.287551 3454 raft_consensus.cc:2781] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:17:09.287963 3454 raft_consensus.cc:686] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63 [term 1 LEADER]: Becoming Leader. State: Replica: 67b9089a4cb642d682558e47961dcc63, State: Running, Role: LEADER
I20260929 20:17:09.288267 3454 consensus_queue.cc:237] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100246ec987b437eb04e30cf2815bced" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33241 } } peers { permanent_uuid: "67b9089a4cb642d682558e47961dcc63" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45841 } } peers { permanent_uuid: "eaa3fccfe54b400e80a3f4c5c33a1baf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42717 } }
I20260929 20:17:09.289055 3455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:09.288583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.27.185.67:57131)
0929 20:17:09.288744 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:09.288864 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 20:17:09.291281 2889 catalog_manager.cc:5495] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67b9089a4cb642d682558e47961dcc63 (127.27.185.67). New cstate: current_term: 1 leader_uuid: "67b9089a4cb642d682558e47961dcc63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100246ec987b437eb04e30cf2815bced" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67b9089a4cb642d682558e47961dcc63" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaa3fccfe54b400e80a3f4c5c33a1baf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42717 } health_report { overall_health: UNKNOWN } } }
I20260929 20:17:09.310730 3448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:09.306083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41874 (local address 127.27.185.67:45841)
0929 20:17:09.306270 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:09.306275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:09.306292 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:17:09.306402 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:09.306406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:09.306483 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:09.306557 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:17:09.306969 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:09.307744 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:09.309155 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:09.309582 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:09.309675 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:09.310067 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:17:09.310166 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:17:09.310228 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:09.310356 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 20:17:09.310419 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:17:09.313902 3022 raft_consensus.cc:1261] T cb82a02976de4814ac2a73ac5db0aa20 P eaa3fccfe54b400e80a3f4c5c33a1baf [term 1 FOLLOWER]: Refusing update from remote peer 67b9089a4cb642d682558e47961dcc63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:17:09.313959 3168 raft_consensus.cc:1261] T cb82a02976de4814ac2a73ac5db0aa20 P 100246ec987b437eb04e30cf2815bced [term 1 FOLLOWER]: Refusing update from remote peer 67b9089a4cb642d682558e47961dcc63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:17:09.314604 3464 consensus_queue.cc:1035] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaa3fccfe54b400e80a3f4c5c33a1baf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:17:09.314998 3454 consensus_queue.cc:1035] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100246ec987b437eb04e30cf2815bced" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 33241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:17:09.317478 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:09.316724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.27.185.65:36145)
0929 20:17:09.317221 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:09.317320 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389}
I20260929 20:17:09.317814 3460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:09.317307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.27.185.66:44771)
0929 20:17:09.317482 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:09.317615 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 20:17:09.324983 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:09.324467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.27.185.65:55705)
0929 20:17:09.324672 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:09.324784 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260929 20:17:09.327109 3455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:09.326657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.27.185.67:41707)
0929 20:17:09.326836 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:09.326944 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 20:17:09.333138 2889 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:17:09.333513 2889 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:17:09.337218 3455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:09.336799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.27.185.67:56073)
0929 20:17:09.336955 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:09.337048 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 20:17:09.341040 3460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:09.340585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.27.185.66:39109)
0929 20:17:09.340807 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:09.340900 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260929 20:17:09.345022 2889 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 927955a2d44a4c7198c39f371d6671d3: Sending DeleteTablet for 3 replicas of tablet cb82a02976de4814ac2a73ac5db0aa20
I20260929 20:17:09.345208 3289 tablet_service.cc:1514] Processing DeleteTablet for tablet cb82a02976de4814ac2a73ac5db0aa20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:17:09 UTC) from {username='oryx', principal='oryx/127.27.185.124@KRBTEST.COM'} at 127.0.0.1:41862
I20260929 20:17:09.345556 3148 tablet_service.cc:1514] Processing DeleteTablet for tablet cb82a02976de4814ac2a73ac5db0aa20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:17:09 UTC) from {username='oryx', principal='oryx/127.27.185.124@KRBTEST.COM'} at 127.0.0.1:39526
I20260929 20:17:09.345937 3476 tablet_replica.cc:330] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63: stopping tablet replica
I20260929 20:17:09.346158 3476 raft_consensus.cc:2229] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:17:09.346431 3002 tablet_service.cc:1514] Processing DeleteTablet for tablet cb82a02976de4814ac2a73ac5db0aa20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:17:09 UTC) from {username='oryx', principal='oryx/127.27.185.124@KRBTEST.COM'} at 127.0.0.1:39486
I20260929 20:17:09.346614 3476 raft_consensus.cc:2258] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:17:09.346715 3477 tablet_replica.cc:330] T cb82a02976de4814ac2a73ac5db0aa20 P 100246ec987b437eb04e30cf2815bced: stopping tablet replica
I20260929 20:17:09.346917 3477 raft_consensus.cc:2229] T cb82a02976de4814ac2a73ac5db0aa20 P 100246ec987b437eb04e30cf2815bced [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:17:09.346966 3288 tablet_service.cc:1514] Processing DeleteTablet for tablet cb82a02976de4814ac2a73ac5db0aa20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:17:09 UTC) from {username='oryx', principal='oryx/127.27.185.124@KRBTEST.COM'} at 127.0.0.1:41862
I20260929 20:17:09.347281 3477 raft_consensus.cc:2258] T cb82a02976de4814ac2a73ac5db0aa20 P 100246ec987b437eb04e30cf2815bced [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:17:09.347901 3476 ts_tablet_manager.cc:1905] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:17:09.348208 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 2855
I20260929 20:17:09.348225 3147 tablet_service.cc:1514] Processing DeleteTablet for tablet cb82a02976de4814ac2a73ac5db0aa20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:17:09 UTC) from {username='oryx', principal='oryx/127.27.185.124@KRBTEST.COM'} at 127.0.0.1:39526
I20260929 20:17:09.348417 3001 tablet_service.cc:1514] Processing DeleteTablet for tablet cb82a02976de4814ac2a73ac5db0aa20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:17:09 UTC) from {username='oryx', principal='oryx/127.27.185.124@KRBTEST.COM'} at 127.0.0.1:39486
I20260929 20:17:09.349785 3477 ts_tablet_manager.cc:1905] T cb82a02976de4814ac2a73ac5db0aa20 P 100246ec987b437eb04e30cf2815bced: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:17:09.357820 3478 tablet_replica.cc:330] T cb82a02976de4814ac2a73ac5db0aa20 P eaa3fccfe54b400e80a3f4c5c33a1baf: stopping tablet replica
I20260929 20:17:09.358053 3478 raft_consensus.cc:2229] T cb82a02976de4814ac2a73ac5db0aa20 P eaa3fccfe54b400e80a3f4c5c33a1baf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:17:09.358355 3478 raft_consensus.cc:2258] T cb82a02976de4814ac2a73ac5db0aa20 P eaa3fccfe54b400e80a3f4c5c33a1baf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:17:09.359047 3476 ts_tablet_manager.cc:1918] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:17:09.359192 3476 log.cc:1196] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-2/wal/wals/cb82a02976de4814ac2a73ac5db0aa20
I20260929 20:17:09.359689 3476 ts_tablet_manager.cc:1939] T cb82a02976de4814ac2a73ac5db0aa20 P 67b9089a4cb642d682558e47961dcc63: Deleting consensus metadata
I20260929 20:17:09.359790 3478 ts_tablet_manager.cc:1905] T cb82a02976de4814ac2a73ac5db0aa20 P eaa3fccfe54b400e80a3f4c5c33a1baf: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 20:17:09.361431 2873 catalog_manager.cc:4808] TS 67b9089a4cb642d682558e47961dcc63 (127.27.185.67:45841): delete failed for tablet cb82a02976de4814ac2a73ac5db0aa20 because the tablet was not found. No further retry: Not found: Tablet not found: cb82a02976de4814ac2a73ac5db0aa20
I20260929 20:17:09.363502 3477 ts_tablet_manager.cc:1918] T cb82a02976de4814ac2a73ac5db0aa20 P 100246ec987b437eb04e30cf2815bced: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:17:09.363765 3477 log.cc:1196] T cb82a02976de4814ac2a73ac5db0aa20 P 100246ec987b437eb04e30cf2815bced: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-1/wal/wals/cb82a02976de4814ac2a73ac5db0aa20
I20260929 20:17:09.364297 3477 ts_tablet_manager.cc:1939] T cb82a02976de4814ac2a73ac5db0aa20 P 100246ec987b437eb04e30cf2815bced: Deleting consensus metadata
I20260929 20:17:09.365716 3478 ts_tablet_manager.cc:1918] T cb82a02976de4814ac2a73ac5db0aa20 P eaa3fccfe54b400e80a3f4c5c33a1baf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:17:09.365866 3478 log.cc:1196] T cb82a02976de4814ac2a73ac5db0aa20 P eaa3fccfe54b400e80a3f4c5c33a1baf: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790712966917858-28389-0/minicluster-data/ts-0/wal/wals/cb82a02976de4814ac2a73ac5db0aa20
I20260929 20:17:09.366400 3478 ts_tablet_manager.cc:1939] T cb82a02976de4814ac2a73ac5db0aa20 P eaa3fccfe54b400e80a3f4c5c33a1baf: Deleting consensus metadata
W20260929 20:17:09.366496 2874 catalog_manager.cc:4808] TS 100246ec987b437eb04e30cf2815bced (127.27.185.66:33241): delete failed for tablet cb82a02976de4814ac2a73ac5db0aa20 because the tablet was not found. No further retry: Not found: Tablet not found: cb82a02976de4814ac2a73ac5db0aa20
W20260929 20:17:09.367880 2876 catalog_manager.cc:4808] TS eaa3fccfe54b400e80a3f4c5c33a1baf (127.27.185.65:42717): delete failed for tablet cb82a02976de4814ac2a73ac5db0aa20 because the tablet was not found. No further retry: Not found: Tablet not found: cb82a02976de4814ac2a73ac5db0aa20
I20260929 20:17:09.370066 3450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:09.349800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39490 (local address 127.27.185.65:42717)
0929 20:17:09.350025 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:09.350032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:09.351465 (+ 1433us) server_negotiation.cc:407] Connection header received
0929 20:17:09.351746 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:09.351752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:09.351838 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:09.351908 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:09.353824 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:09.354974 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:09.357314 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:09.358161 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:09.358262 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:09.362749 (+ 4487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:09.362773 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:09.362794 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:09.362836 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:09.367129 (+ 4293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:09.368129 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:09.368137 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:09.368139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:09.368279 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:09.368694 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:09.368697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:09.368699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:09.369043 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:09.369258 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:09.369621 (+ 363us) server_negotiation.cc:299] Negotiation successful
0929 20:17:09.369798 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:17:09.370898 3062 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:09.455240 3431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:09.454826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.0.0.1:35020)
0929 20:17:09.454997 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:09.455093 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 20:17:09.456322 3431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:09.455953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.0.0.1:35022)
0929 20:17:09.456112 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:09.456199 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 20:17:09.483446 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 2855
I20260929 20:17:09.501107 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 3079
I20260929 20:17:09.515511 3447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:09.501953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39536 (local address 127.27.185.66:33241)
0929 20:17:09.502167 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:09.502173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:09.502193 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:17:09.502386 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:09.502392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:09.502481 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:09.502593 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:09.503993 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:09.504812 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:09.506084 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:09.506490 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:09.506632 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:09.509219 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:09.509246 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:09.509263 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:09.509310 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:09.512666 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:09.513343 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:09.513353 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:09.513357 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:09.513486 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:09.513872 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:09.513879 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:09.513882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:09.514232 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:09.514549 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:09.514840 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 20:17:09.515087 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:17:09.516227 3208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:09.610977 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 3079
I20260929 20:17:09.629040 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 3226
I20260929 20:17:09.642130 3448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:09.629825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41890 (local address 127.27.185.67:45841)
0929 20:17:09.630021 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:09.630026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:09.630043 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:17:09.630208 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:09.630212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:09.630273 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:09.630353 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:09.631773 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:09.632474 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:09.633626 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:09.634009 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:09.634098 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:09.636605 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:09.636623 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:09.636639 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:09.636674 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:09.639720 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:09.640313 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:09.640321 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:09.640323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:09.640457 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:09.640765 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:09.640768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:09.640770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:09.641126 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:09.641346 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:09.641584 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 20:17:09.641755 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:17:09.642845 3349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:09.765661 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 3226
I20260929 20:17:09.787505 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 2564
I20260929 20:17:09.788632 2731 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:09.865456 3431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:09.864973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.0.0.1:35026)
0929 20:17:09.865167 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:09.865280 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 20:17:09.866532 3431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:09.866149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36737 (local address 127.0.0.1:35036)
0929 20:17:09.866304 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:09.866392 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:36737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 20:17:09.909337 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 2564
I20260929 20:17:09.929764 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 2664
I20260929 20:17:09.930828 2919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:10.053951 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 2664
2026-09-29T20:17:10Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9887 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:17:10 dist-test-slave-4cx3 krb5kdc[3487](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:17:10 dist-test-slave-4cx3 krb5kdc[3487](info): set up 2 sockets
Sep 29 20:17:10 dist-test-slave-4cx3 krb5kdc[3487](info): commencing operation
krb5kdc: starting...
W20260929 20:17:11.183583 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:17:11 dist-test-slave-4cx3 krb5kdc[3487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:17:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:17:11Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.126.keytab.
Entry for principal oryx/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.126.keytab.
I20260929 20:17:11.407756 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:45075
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.126.keytab
--principal=oryx/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:44241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:45075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:17:11.739991 3500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:11.740288 3500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:11.740381 3500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:11.747743 3500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:17:11.747854 3500 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 20:17:11.747896 3500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:11.747958 3500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:11.748011 3500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:17:11.748068 3500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:17:11.758189 3500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:44241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:45075
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.126.keytab
--principal=oryx/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:45075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:11.759560 3500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:11.760860 3500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:11.773366 3508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:11.773394 3506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:11.773618 3505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:11.774271 3500 server_base.cc:1034] running on GCE node
I20260929 20:17:11.775115 3500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:11.776715 3500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:11.777968 3500 hybrid_clock.cc:648] HybridClock initialized: now 1790713031777898 us; error 48 us; skew 500 ppm
I20260929 20:17:11.778326 3500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:17:11 dist-test-slave-4cx3 krb5kdc[3487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713031, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:17:11.784420 3500 init.cc:377] Logged in from keytab as oryx/127.27.185.126@KRBTEST.COM (short username oryx)
I20260929 20:17:11.786049 3500 webserver.cc:460] Webserver started at http://127.27.185.126:43763/ using document root <none> and password file <none>
I20260929 20:17:11.786782 3500 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:11.786880 3500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:11.787159 3500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:17:11.790330 3500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "424e7e4d77534830b1f0ac96ba2e4982"
format_stamp: "Formatted at 2026-09-29 20:17:11 on dist-test-slave-4cx3"
server_key: "b778cc750eb43a48384cd92475a4664c"
server_key_iv: "5f351234b51c9f21557cb9b22e1dafb0"
server_key_version: "encryptionkey@0"
I20260929 20:17:11.791095 3500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "424e7e4d77534830b1f0ac96ba2e4982"
format_stamp: "Formatted at 2026-09-29 20:17:11 on dist-test-slave-4cx3"
server_key: "b778cc750eb43a48384cd92475a4664c"
server_key_iv: "5f351234b51c9f21557cb9b22e1dafb0"
server_key_version: "encryptionkey@0"
I20260929 20:17:11.796563 3500 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 20:17:11.800225 3515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:11.801626 3500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:17:11.801811 3500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "424e7e4d77534830b1f0ac96ba2e4982"
format_stamp: "Formatted at 2026-09-29 20:17:11 on dist-test-slave-4cx3"
server_key: "b778cc750eb43a48384cd92475a4664c"
server_key_iv: "5f351234b51c9f21557cb9b22e1dafb0"
server_key_version: "encryptionkey@0"
I20260929 20:17:11.802052 3500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:11.831058 3500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:11.835199 3500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:11.835526 3500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:11.857764 3500 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:45075
I20260929 20:17:11.857784 3576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:45075 every 8 connection(s)
I20260929 20:17:11.859160 3500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:17:11.863045 3577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:11.869261 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 3500
I20260929 20:17:11.869542 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:17:11.870352 28389 external_mini_cluster.cc:1428] Setting key 9d52e65f249e10621266f30e5f8e4c66
I20260929 20:17:11.874878 3577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 424e7e4d77534830b1f0ac96ba2e4982: Bootstrap starting.
I20260929 20:17:11.878707 3577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 424e7e4d77534830b1f0ac96ba2e4982: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:11.880035 3577 log.cc:824] T 00000000000000000000000000000000 P 424e7e4d77534830b1f0ac96ba2e4982: Log is configured to *not* fsync() on all Append() calls
I20260929 20:17:11.883203 3577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 424e7e4d77534830b1f0ac96ba2e4982: No bootstrap required, opened a new log
Sep 29 20:17:11 dist-test-slave-4cx3 krb5kdc[3487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM
I20260929 20:17:11.891371 3577 raft_consensus.cc:348] T 00000000000000000000000000000000 P 424e7e4d77534830b1f0ac96ba2e4982 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "424e7e4d77534830b1f0ac96ba2e4982" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 45075 } }
I20260929 20:17:11.891829 3577 raft_consensus.cc:374] T 00000000000000000000000000000000 P 424e7e4d77534830b1f0ac96ba2e4982 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:11.891954 3577 raft_consensus.cc:729] T 00000000000000000000000000000000 P 424e7e4d77534830b1f0ac96ba2e4982 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 424e7e4d77534830b1f0ac96ba2e4982, State: Initialized, Role: FOLLOWER
I20260929 20:17:11.892655 3577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 424e7e4d77534830b1f0ac96ba2e4982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "424e7e4d77534830b1f0ac96ba2e4982" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 45075 } }
I20260929 20:17:11.892964 3577 raft_consensus.cc:388] T 00000000000000000000000000000000 P 424e7e4d77534830b1f0ac96ba2e4982 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:17:11.893124 3577 raft_consensus.cc:482] T 00000000000000000000000000000000 P 424e7e4d77534830b1f0ac96ba2e4982 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:17:11.893314 3577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 424e7e4d77534830b1f0ac96ba2e4982 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:17:11.894048 3580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:11.872574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55392 (local address 127.27.185.126:45075)
0929 20:17:11.873408 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:11.873420 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:11.873461 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:17:11.875167 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:11.875190 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:11.875659 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:11.875980 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:11.877815 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:11.879179 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:11.880928 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:11.881748 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:11.881875 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:11.886506 (+ 4631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:11.886513 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:11.886528 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:11.886563 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:11.889767 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:11.890473 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:11.890480 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:11.890486 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:11.890667 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:11.891065 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:11.891070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:11.891072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:11.891661 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:11.891965 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:11.892386 (+ 421us) server_negotiation.cc:299] Negotiation successful
0929 20:17:11.892810 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":408,"threads_started":1}
I20260929 20:17:11.895843 3577 raft_consensus.cc:504] T 00000000000000000000000000000000 P 424e7e4d77534830b1f0ac96ba2e4982 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "424e7e4d77534830b1f0ac96ba2e4982" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 45075 } }
I20260929 20:17:11.896359 3577 leader_election.cc:302] T 00000000000000000000000000000000 P 424e7e4d77534830b1f0ac96ba2e4982 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 424e7e4d77534830b1f0ac96ba2e4982; no voters:
I20260929 20:17:11.897044 3577 leader_election.cc:288] T 00000000000000000000000000000000 P 424e7e4d77534830b1f0ac96ba2e4982 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:17:11.897113 3582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 424e7e4d77534830b1f0ac96ba2e4982 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:17:11.897902 3582 raft_consensus.cc:686] T 00000000000000000000000000000000 P 424e7e4d77534830b1f0ac96ba2e4982 [term 1 LEADER]: Becoming Leader. State: Replica: 424e7e4d77534830b1f0ac96ba2e4982, State: Running, Role: LEADER
I20260929 20:17:11.898550 3582 consensus_queue.cc:237] T 00000000000000000000000000000000 P 424e7e4d77534830b1f0ac96ba2e4982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "424e7e4d77534830b1f0ac96ba2e4982" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 45075 } }
I20260929 20:17:11.899466 3577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 424e7e4d77534830b1f0ac96ba2e4982 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:17:11.901861 3583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 424e7e4d77534830b1f0ac96ba2e4982 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "424e7e4d77534830b1f0ac96ba2e4982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "424e7e4d77534830b1f0ac96ba2e4982" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 45075 } } }
I20260929 20:17:11.902309 3583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 424e7e4d77534830b1f0ac96ba2e4982 [sys.catalog]: This master's current role is: LEADER
I20260929 20:17:11.901832 3584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 424e7e4d77534830b1f0ac96ba2e4982 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 424e7e4d77534830b1f0ac96ba2e4982. Latest consensus state: current_term: 1 leader_uuid: "424e7e4d77534830b1f0ac96ba2e4982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "424e7e4d77534830b1f0ac96ba2e4982" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 45075 } } }
I20260929 20:17:11.902688 3584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 424e7e4d77534830b1f0ac96ba2e4982 [sys.catalog]: This master's current role is: LEADER
I20260929 20:17:11.911952 3590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:17:11.917932 3590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:17:11.926216 3590 catalog_manager.cc:1347] Generated new cluster ID: 2a62d0fcbb144a2897ecb0b170c9105a
I20260929 20:17:11.926342 3590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:17:11.941908 3590 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:17:11.944105 3590 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:17:11.963162 3590 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 424e7e4d77534830b1f0ac96ba2e4982: Generated new TSK 0
I20260929 20:17:11.964363 3590 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.65.keytab.
Entry for principal oryx/127.27.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.65.keytab.
I20260929 20:17:12.025123 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:0
--local_ip_for_outbound_sockets=127.27.185.65
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:45075
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.65.keytab
--principal=oryx/127.27.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:44241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:17:12.400179 3603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:12.400534 3603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:12.400699 3603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:12.408182 3603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:12.408334 3603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:12.408475 3603 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:17:12.408612 3603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:17:12.419191 3603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:44241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.65.keytab
--principal=oryx/127.27.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.185.126:45075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:12.420604 3603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:12.421975 3603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:12.435591 3609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:12.435637 3611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:12.435669 3608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:12.436504 3603 server_base.cc:1034] running on GCE node
I20260929 20:17:12.437067 3603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:12.437978 3603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:12.439199 3603 hybrid_clock.cc:648] HybridClock initialized: now 1790713032439140 us; error 30 us; skew 500 ppm
I20260929 20:17:12.439563 3603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:17:12 dist-test-slave-4cx3 krb5kdc[3487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713032, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:17:12.442727 3603 init.cc:377] Logged in from keytab as oryx/127.27.185.65@KRBTEST.COM (short username oryx)
I20260929 20:17:12.444334 3603 webserver.cc:460] Webserver started at http://127.27.185.65:43563/ using document root <none> and password file <none>
I20260929 20:17:12.445101 3603 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:12.445218 3603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:12.445561 3603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:17:12.448675 3603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-0/data/instance:
uuid: "57b2ac153bca44e9b3e4e5c82e3eb771"
format_stamp: "Formatted at 2026-09-29 20:17:12 on dist-test-slave-4cx3"
server_key: "935d8322fe34389e2927a3c2669affe2"
server_key_iv: "44d71338193f8fc3546c89a111cb80b5"
server_key_version: "encryptionkey@0"
I20260929 20:17:12.449508 3603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance:
uuid: "57b2ac153bca44e9b3e4e5c82e3eb771"
format_stamp: "Formatted at 2026-09-29 20:17:12 on dist-test-slave-4cx3"
server_key: "935d8322fe34389e2927a3c2669affe2"
server_key_iv: "44d71338193f8fc3546c89a111cb80b5"
server_key_version: "encryptionkey@0"
I20260929 20:17:12.454722 3603 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:17:12.458360 3618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:12.459801 3603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 20:17:12.459990 3603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "57b2ac153bca44e9b3e4e5c82e3eb771"
format_stamp: "Formatted at 2026-09-29 20:17:12 on dist-test-slave-4cx3"
server_key: "935d8322fe34389e2927a3c2669affe2"
server_key_iv: "44d71338193f8fc3546c89a111cb80b5"
server_key_version: "encryptionkey@0"
I20260929 20:17:12.460216 3603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:12.485680 3603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:12.490016 3603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:12.490306 3603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:12.493296 3603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:17:12.493405 3603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:12.493539 3603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:17:12.493628 3603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:17:12 dist-test-slave-4cx3 krb5kdc[3487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713032, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM
I20260929 20:17:12.519804 3627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:12.493827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:37871 (local address 127.27.185.126:45075)
0929 20:17:12.494651 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:12.494655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:12.495837 (+ 1182us) server_negotiation.cc:407] Connection header received
0929 20:17:12.497922 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:12.497927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:12.498062 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:12.498195 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:12.501158 (+ 2963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:12.502111 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:12.505543 (+ 3432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:12.506008 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:12.506117 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:12.512259 (+ 6142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:12.512269 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:12.512277 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:12.512449 (+ 172us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:12.516345 (+ 3896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:12.517368 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:12.517374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:12.517377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:12.517509 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:12.517939 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:12.517943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:12.517947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:12.518227 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:12.518385 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:12.519151 (+ 766us) server_negotiation.cc:299] Negotiation successful
0929 20:17:12.519373 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":629,"threads_started":1}
I20260929 20:17:12.520920 3628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:12.494572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:45075 (local address 127.27.185.65:37871)
0929 20:17:12.495652 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:12.495702 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:12.497578 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:12.498535 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:12.498552 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:12.499712 (+ 1160us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:12.500915 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:12.500945 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:12.502330 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:12.502340 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:12.505291 (+ 2951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:12.505313 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:12.506292 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:12.506303 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:12.506691 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:12.507693 (+ 1002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:12.507740 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:12.511793 (+ 4053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:12.516482 (+ 4689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:12.516499 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:12.516517 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:12.517243 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:12.517627 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:12.517632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:12.517635 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:12.517822 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:12.518360 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:12.518369 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:12.518742 (+ 373us) client_negotiation.cc:769] Sending connection context
0929 20:17:12.519037 (+ 295us) client_negotiation.cc:240] Negotiation successful
0929 20:17:12.519622 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"thread_start_us":480,"threads_started":1}
I20260929 20:17:12.542286 3673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:12.527267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:45075 (local address 127.27.185.65:37797)
0929 20:17:12.528041 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:12.528068 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:12.528245 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:12.528577 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:12.528584 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:12.529347 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:12.530018 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:12.530038 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:12.531054 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:12.531061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:12.532692 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:12.532708 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:12.533873 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:12.533878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:12.534077 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:12.534980 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:12.535008 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:12.536143 (+ 1135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:12.539538 (+ 3395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:12.539543 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:12.539550 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:12.540057 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:12.540559 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:12.540563 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:12.540566 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:12.540693 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:12.541448 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:12.541451 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:12.541609 (+ 158us) client_negotiation.cc:769] Sending connection context
0929 20:17:12.541712 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 20:17:12.541958 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":530,"threads_started":1}
I20260929 20:17:12.542714 3627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:12.527639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:37797 (local address 127.27.185.126:45075)
0929 20:17:12.527790 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:12.527794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:12.528168 (+ 374us) server_negotiation.cc:407] Connection header received
0929 20:17:12.528366 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:12.528370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:12.528433 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:12.528512 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:12.530201 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:12.530896 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:12.532856 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:12.533429 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:12.533596 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:12.536382 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:12.536387 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:12.536390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:12.536421 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:12.539364 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:12.540223 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:12.540231 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:12.540237 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:12.540423 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:12.540819 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:12.540827 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:12.540832 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:12.541315 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:12.541498 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:12.541894 (+ 396us) server_negotiation.cc:299] Negotiation successful
0929 20:17:12.542222 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 20:17:12.556730 3603 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:43463
I20260929 20:17:12.556754 3738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:43463 every 8 connection(s)
I20260929 20:17:12.558161 3603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:17:12.560451 3739 heartbeater.cc:347] Connected to a master server at 127.27.185.126:45075
I20260929 20:17:12.560672 3739 heartbeater.cc:464] Registering TS with master...
I20260929 20:17:12.561295 3739 heartbeater.cc:511] Master 127.27.185.126:45075 requested a full tablet report, sending...
I20260929 20:17:12.563316 3532 ts_manager.cc:194] Registered new tserver with Master: 57b2ac153bca44e9b3e4e5c82e3eb771 (127.27.185.65:43463)
I20260929 20:17:12.564993 3532 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.185.65@KRBTEST.COM'} at 127.27.185.65:37871
I20260929 20:17:12.568596 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 3603
I20260929 20:17:12.568781 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance
I20260929 20:17:12.569379 28389 external_mini_cluster.cc:1428] Setting key b977a908d41e12b4030d89e84cb0d5c8
WARNING: no policy specified for oryx/127.27.185.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.185.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.66.keytab.
Entry for principal oryx/127.27.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.66.keytab.
I20260929 20:17:12.618337 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:0
--local_ip_for_outbound_sockets=127.27.185.66
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:45075
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.66.keytab
--principal=oryx/127.27.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:44241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:17:12.953666 3743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:12.954030 3743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:12.954202 3743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:12.961454 3743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:12.961622 3743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:12.961779 3743 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:17:12.961917 3743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:17:12.972290 3743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:44241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.66.keytab
--principal=oryx/127.27.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.185.126:45075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:12.973611 3743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:12.974951 3743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:12.988114 3749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:12.988154 3748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:12.988128 3751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:12.989068 3743 server_base.cc:1034] running on GCE node
I20260929 20:17:12.989753 3743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:12.990684 3743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:12.991932 3743 hybrid_clock.cc:648] HybridClock initialized: now 1790713032991848 us; error 59 us; skew 500 ppm
I20260929 20:17:12.992251 3743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:17:12 dist-test-slave-4cx3 krb5kdc[3487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713032, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:17:12.995472 3743 init.cc:377] Logged in from keytab as oryx/127.27.185.66@KRBTEST.COM (short username oryx)
I20260929 20:17:12.996930 3743 webserver.cc:460] Webserver started at http://127.27.185.66:39597/ using document root <none> and password file <none>
I20260929 20:17:12.997609 3743 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:12.997697 3743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:12.997957 3743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:17:13.001228 3743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-1/data/instance:
uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd"
format_stamp: "Formatted at 2026-09-29 20:17:12 on dist-test-slave-4cx3"
server_key: "4e10223369240abe17488a3efcc09267"
server_key_iv: "fc69db94851aa8a9f8873431f639e6f7"
server_key_version: "encryptionkey@0"
I20260929 20:17:13.001997 3743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance:
uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd"
format_stamp: "Formatted at 2026-09-29 20:17:12 on dist-test-slave-4cx3"
server_key: "4e10223369240abe17488a3efcc09267"
server_key_iv: "fc69db94851aa8a9f8873431f639e6f7"
server_key_version: "encryptionkey@0"
I20260929 20:17:13.007278 3743 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:17:13.010860 3758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:13.012367 3743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:17:13.012533 3743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd"
format_stamp: "Formatted at 2026-09-29 20:17:12 on dist-test-slave-4cx3"
server_key: "4e10223369240abe17488a3efcc09267"
server_key_iv: "fc69db94851aa8a9f8873431f639e6f7"
server_key_version: "encryptionkey@0"
I20260929 20:17:13.012746 3743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:13.024658 3743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:13.028981 3743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:13.029274 3743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:13.032270 3743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:17:13.032395 3743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:13.032588 3743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:17:13.032689 3743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:17:13 dist-test-slave-4cx3 krb5kdc[3487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713032, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM
I20260929 20:17:13.058923 3767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:13.033382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:45075 (local address 127.27.185.66:38745)
0929 20:17:13.034179 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:13.034217 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:13.036167 (+ 1950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:13.036972 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:13.036987 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:13.037873 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:13.038760 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:13.038784 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.040546 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.040551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:13.042753 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:13.042771 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.043987 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.044011 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:13.044366 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:13.045213 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:13.045256 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:13.048825 (+ 3569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:13.053465 (+ 4640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:13.053474 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:13.053505 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:13.054111 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:13.054625 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:13.054636 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:13.054641 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:13.054966 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:13.055629 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:13.055637 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:13.056203 (+ 566us) client_negotiation.cc:769] Sending connection context
0929 20:17:13.056525 (+ 322us) client_negotiation.cc:240] Negotiation successful
0929 20:17:13.057151 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":354,"threads_started":1}
I20260929 20:17:13.060165 3627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:13.032680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:38745 (local address 127.27.185.126:45075)
0929 20:17:13.032850 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:13.032855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:13.034352 (+ 1497us) server_negotiation.cc:407] Connection header received
0929 20:17:13.036467 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:13.036473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:13.036573 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:13.036683 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:13.038995 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.040355 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.042931 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.043697 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.043854 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:13.049191 (+ 5337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:13.049201 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:13.049209 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:13.049268 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:13.053311 (+ 4043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:13.054273 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:13.054282 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:13.054288 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:13.054452 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:13.055106 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:13.055111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:13.055115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:13.055470 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:13.055646 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:13.059592 (+ 3946us) server_negotiation.cc:299] Negotiation successful
0929 20:17:13.059812 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 20:17:13.087838 3743 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:43357
I20260929 20:17:13.087809 3877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:43357 every 8 connection(s)
I20260929 20:17:13.089522 3743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:17:13.090950 3846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:13.065936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:45075 (local address 127.27.185.66:45363)
0929 20:17:13.076153 (+ 10217us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:13.076173 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:13.076404 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:13.076891 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:13.076895 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:13.077391 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:13.078072 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:13.078088 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.079127 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.079134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:13.080758 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:13.080771 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.081688 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.081695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:13.081855 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:13.082521 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:13.082539 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:13.083253 (+ 714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:13.087914 (+ 4661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:13.087921 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:13.087928 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:13.088414 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:13.088812 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:13.088815 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:13.088817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:13.088933 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:13.089650 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:13.089654 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:13.089882 (+ 228us) client_negotiation.cc:769] Sending connection context
0929 20:17:13.090053 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 20:17:13.090401 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10024,"thread_start_us":9772,"threads_started":1}
I20260929 20:17:13.091755 3627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:13.065886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:45363 (local address 127.27.185.126:45075)
0929 20:17:13.066151 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:13.066158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:13.076349 (+ 10191us) server_negotiation.cc:407] Connection header received
0929 20:17:13.076588 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:13.076598 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:13.076719 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:13.076868 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:13.078253 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.078980 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.080922 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.081478 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.081619 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:13.083508 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:13.083514 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:13.083517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:13.083550 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:13.087767 (+ 4217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:13.088542 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:13.088548 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:13.088551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:13.088700 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:13.089044 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:13.089050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:13.089052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:13.089520 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:13.089678 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:13.091159 (+ 1481us) server_negotiation.cc:299] Negotiation successful
0929 20:17:13.091404 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 20:17:13.091817 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 3743
I20260929 20:17:13.092006 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance
I20260929 20:17:13.092425 28389 external_mini_cluster.cc:1428] Setting key 643a0819430e20943d62a014d6eab84d
I20260929 20:17:13.092967 3878 heartbeater.cc:347] Connected to a master server at 127.27.185.126:45075
I20260929 20:17:13.093242 3878 heartbeater.cc:464] Registering TS with master...
I20260929 20:17:13.094020 3878 heartbeater.cc:511] Master 127.27.185.126:45075 requested a full tablet report, sending...
I20260929 20:17:13.095580 3531 ts_manager.cc:194] Registered new tserver with Master: b7a9401c551f4b2aa3bb0d32c39a4abd (127.27.185.66:43357)
I20260929 20:17:13.096451 3531 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.185.66@KRBTEST.COM'} at 127.27.185.66:38745
WARNING: no policy specified for oryx/127.27.185.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.67.keytab.
Entry for principal oryx/127.27.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.67.keytab.
I20260929 20:17:13.140673 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:0
--local_ip_for_outbound_sockets=127.27.185.67
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:45075
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.67.keytab
--principal=oryx/127.27.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:44241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:17:13.470479 3882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:13.470813 3882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:13.470994 3882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:13.478391 3882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:13.478549 3882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:13.478705 3882 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:17:13.478845 3882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:17:13.488895 3882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:44241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.67.keytab
--principal=oryx/127.27.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.185.126:45075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:13.490214 3882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:13.491598 3882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:13.504474 3887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:13.504500 3890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:13.504556 3888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:13.505748 3882 server_base.cc:1034] running on GCE node
I20260929 20:17:13.506438 3882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:13.507375 3882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:13.508670 3882 hybrid_clock.cc:648] HybridClock initialized: now 1790713033508610 us; error 36 us; skew 500 ppm
I20260929 20:17:13.508945 3882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:17:13 dist-test-slave-4cx3 krb5kdc[3487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713033, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:17:13.512247 3882 init.cc:377] Logged in from keytab as oryx/127.27.185.67@KRBTEST.COM (short username oryx)
I20260929 20:17:13.513830 3882 webserver.cc:460] Webserver started at http://127.27.185.67:35433/ using document root <none> and password file <none>
I20260929 20:17:13.514521 3882 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:13.514632 3882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:13.514961 3882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:17:13.517969 3882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-2/data/instance:
uuid: "5deeb3b1231942048cd724c8f314a9c1"
format_stamp: "Formatted at 2026-09-29 20:17:13 on dist-test-slave-4cx3"
server_key: "00f3fdeb151e0db227ba2be16fa683ce"
server_key_iv: "ab7db88294b52cc4e8455766f1743b73"
server_key_version: "encryptionkey@0"
I20260929 20:17:13.518752 3882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance:
uuid: "5deeb3b1231942048cd724c8f314a9c1"
format_stamp: "Formatted at 2026-09-29 20:17:13 on dist-test-slave-4cx3"
server_key: "00f3fdeb151e0db227ba2be16fa683ce"
server_key_iv: "ab7db88294b52cc4e8455766f1743b73"
server_key_version: "encryptionkey@0"
I20260929 20:17:13.523962 3882 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:17:13.527489 3897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:13.528796 3882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:17:13.528980 3882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "5deeb3b1231942048cd724c8f314a9c1"
format_stamp: "Formatted at 2026-09-29 20:17:13 on dist-test-slave-4cx3"
server_key: "00f3fdeb151e0db227ba2be16fa683ce"
server_key_iv: "ab7db88294b52cc4e8455766f1743b73"
server_key_version: "encryptionkey@0"
I20260929 20:17:13.529211 3882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:13.540753 3882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:13.544544 3882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:13.544811 3882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:13.547333 3882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:17:13.547461 3882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 20:17:13.547623 3882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:17:13.547700 3882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:17:13 dist-test-slave-4cx3 krb5kdc[3487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713033, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM
I20260929 20:17:13.568105 3739 heartbeater.cc:503] Master 127.27.185.126:45075 was elected leader, sending a full tablet report...
I20260929 20:17:13.573065 3627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:13.548295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:57641 (local address 127.27.185.126:45075)
0929 20:17:13.548486 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:13.548493 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:13.549903 (+ 1410us) server_negotiation.cc:407] Connection header received
0929 20:17:13.552248 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:13.552256 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:13.552376 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:13.552506 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:13.554831 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.556001 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.558692 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.559350 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.559482 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:13.564550 (+ 5068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:13.564560 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:13.564566 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:13.564626 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:13.568879 (+ 4253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:13.570114 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:13.570121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:13.570127 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:13.570278 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:13.570749 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:13.570755 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:13.570758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:13.571054 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:13.571224 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:13.572449 (+ 1225us) server_negotiation.cc:299] Negotiation successful
0929 20:17:13.572702 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:17:13.574564 3906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:13.548824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:45075 (local address 127.27.185.67:57641)
0929 20:17:13.549703 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:13.549761 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:13.551878 (+ 2117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:13.552820 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:13.552837 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:13.553651 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:13.554560 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:13.554595 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.556188 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.556193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:13.558501 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:13.558516 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.559835 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.559843 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:13.560145 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:13.560933 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:13.560968 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:13.564281 (+ 3313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:13.569146 (+ 4865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:13.569160 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:13.569177 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:13.569907 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:13.570409 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:13.570414 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:13.570418 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:13.570631 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:13.571332 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:13.571344 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:13.571957 (+ 613us) client_negotiation.cc:769] Sending connection context
0929 20:17:13.572333 (+ 376us) client_negotiation.cc:240] Negotiation successful
0929 20:17:13.572928 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":415,"threads_started":1}
I20260929 20:17:13.596156 4016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:33609 every 8 connection(s)
I20260929 20:17:13.596156 3882 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:33609
I20260929 20:17:13.597913 3882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:17:13.600189 4017 heartbeater.cc:347] Connected to a master server at 127.27.185.126:45075
I20260929 20:17:13.600449 4017 heartbeater.cc:464] Registering TS with master...
I20260929 20:17:13.600723 3627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:13.582146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:43517 (local address 127.27.185.126:45075)
0929 20:17:13.582384 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:13.582392 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:13.583349 (+ 957us) server_negotiation.cc:407] Connection header received
0929 20:17:13.583605 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:13.583610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:13.583736 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:13.583850 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:13.585357 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.586267 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.588753 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.589603 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.589762 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:13.593233 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:13.593242 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:13.593248 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:13.593302 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:13.597131 (+ 3829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:13.597793 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:13.597799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:13.597803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:13.597973 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:13.598374 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:13.598380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:13.598385 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:13.598830 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:13.599006 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:13.600037 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0929 20:17:13.600268 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:17:13.600890 3986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:13.582313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:45075 (local address 127.27.185.67:43517)
0929 20:17:13.583216 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:13.583243 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:13.583455 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:13.583920 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:13.583932 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:13.584666 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:13.585177 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:13.585193 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.586459 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.586463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:13.588499 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:13.588521 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.590104 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.590115 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:13.590363 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:13.591192 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:13.591224 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:13.592938 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:13.597266 (+ 4328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:13.597270 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:13.597273 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:13.597676 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:13.598114 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:13.598118 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:13.598120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:13.598243 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:13.599075 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:13.599081 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:13.599328 (+ 247us) client_negotiation.cc:769] Sending connection context
0929 20:17:13.599496 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 20:17:13.599844 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":580,"threads_started":1}
I20260929 20:17:13.601318 4017 heartbeater.cc:511] Master 127.27.185.126:45075 requested a full tablet report, sending...
I20260929 20:17:13.602754 3531 ts_manager.cc:194] Registered new tserver with Master: 5deeb3b1231942048cd724c8f314a9c1 (127.27.185.67:33609)
I20260929 20:17:13.603704 3531 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.185.67@KRBTEST.COM'} at 127.27.185.67:57641
I20260929 20:17:13.604403 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 3882
I20260929 20:17:13.604604 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance
I20260929 20:17:13.605132 28389 external_mini_cluster.cc:1428] Setting key 2ad9d7c13f3427980d9001cb458ca9e4
I20260929 20:17:13.607854 28389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 20:17:13 dist-test-slave-4cx3 krb5kdc[3487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713033, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 20:17:13 dist-test-slave-4cx3 krb5kdc[3487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:13 dist-test-slave-4cx3 krb5kdc[3487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 20:17:13.658648 3627 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:13.647909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59298 (local address 127.27.185.126:45075)
0929 20:17:13.648142 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:13.648150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:13.648378 (+ 228us) server_negotiation.cc:407] Connection header received
0929 20:17:13.648599 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:13.648604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:13.648676 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:13.648792 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:13.650155 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.650887 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.652243 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.652678 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.652825 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:13.658292 (+ 5467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59298: BlockingRecv error: recv got EOF from 127.0.0.1:59298 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 29 20:17:13 dist-test-slave-4cx3 krb5kdc[3487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713033, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM
I20260929 20:17:13.675644 3627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:13.661844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59310 (local address 127.27.185.126:45075)
0929 20:17:13.662010 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:13.662014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:13.662455 (+ 441us) server_negotiation.cc:407] Connection header received
0929 20:17:13.662574 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:13.662578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:13.662654 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:13.662758 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:13.663955 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.664831 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.665981 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.666390 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.666483 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:13.670143 (+ 3660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:13.670148 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:13.670151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:13.670186 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:13.673123 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:13.673813 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:13.673817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:13.673819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:13.673950 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:13.674335 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:13.674338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:13.674340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:13.674627 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:13.674865 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:13.675159 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 20:17:13.675328 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 20:17:13.679832 3531 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:17:13.686805 3531 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:17:13.710613 4038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:13.699553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42566 (local address 127.27.185.66:43357)
0929 20:17:13.702219 (+ 2666us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:13.702228 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:13.702256 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:17:13.702354 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:13.702363 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:13.702593 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:13.703511 (+ 918us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:17:13.703690 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.705094 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.708073 (+ 2979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.709143 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.709322 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:13.709560 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:13.710251 (+ 691us) server_negotiation.cc:299] Negotiation successful
0929 20:17:13.710403 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2516,"thread_start_us":2404,"threads_started":1}
I20260929 20:17:13.710776 4039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:13.698751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:43357 (local address 127.0.0.1:42566)
0929 20:17:13.701632 (+ 2881us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:13.701649 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:13.701756 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:13.702805 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:13.702816 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:13.702839 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:17:13.703284 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:13.703296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.705286 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.705293 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:13.707880 (+ 2587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:13.707919 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.709700 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.709709 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:13.709898 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:13.709953 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 20:17:13.710089 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 20:17:13.710313 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2600,"mutex_wait_us":34,"thread_start_us":1717,"threads_started":1}
I20260929 20:17:13.711436 4036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:13.698558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:43463 (local address 127.0.0.1:38230)
0929 20:17:13.700868 (+ 2310us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:13.700902 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:13.701141 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:13.702441 (+ 1300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:13.702448 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:13.702478 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:17:13.702952 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:13.702969 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.705943 (+ 2974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.705948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:13.708440 (+ 2492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:13.708454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.710543 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.710549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:13.710741 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:13.710784 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 20:17:13.710875 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 20:17:13.711009 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2048,"mutex_wait_us":49,"thread_start_us":7366,"threads_started":1}
I20260929 20:17:13.711608 4037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:13.698820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38230 (local address 127.27.185.65:43463)
0929 20:17:13.701763 (+ 2943us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:13.701772 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:13.701801 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:17:13.701931 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:13.701941 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:13.702208 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:13.703214 (+ 1006us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:17:13.703443 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.705093 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.708605 (+ 3512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.710080 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.710210 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:13.710392 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:13.711127 (+ 735us) server_negotiation.cc:299] Negotiation successful
0929 20:17:13.711303 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2753,"thread_start_us":2648,"threads_started":1}
I20260929 20:17:13.715546 4035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:13.700511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:33609 (local address 127.0.0.1:44394)
0929 20:17:13.706266 (+ 5755us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:13.706283 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:13.706401 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:13.707266 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:13.707271 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:13.707302 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:17:13.707768 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:13.707780 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.709684 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.709692 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:13.712690 (+ 2998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:13.712704 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.714713 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.714720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:13.714916 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:13.714950 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 20:17:13.715050 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 20:17:13.715175 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5614,"thread_start_us":884,"threads_started":1}
I20260929 20:17:13.715648 4040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:13.701185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44394 (local address 127.27.185.67:33609)
0929 20:17:13.705708 (+ 4523us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:13.705715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:13.706530 (+ 815us) server_negotiation.cc:407] Connection header received
0929 20:17:13.706627 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:13.706634 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:13.706854 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:13.707038 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:17:13.708018 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.709490 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.712905 (+ 3415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.714459 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.714602 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:13.714857 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:13.715131 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 20:17:13.715301 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4348,"thread_start_us":4152,"threads_started":1}
I20260929 20:17:13.715276 3674 tablet_service.cc:1467] Processing CreateTablet for tablet c642583029ab46e49e51ce1a3b365971 (DEFAULT_TABLE table=test-table [id=f786ac3c8a984b59ad8de9ac0d10b7b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:17:13.715278 3808 tablet_service.cc:1467] Processing CreateTablet for tablet c642583029ab46e49e51ce1a3b365971 (DEFAULT_TABLE table=test-table [id=f786ac3c8a984b59ad8de9ac0d10b7b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:17:13.717034 3808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c642583029ab46e49e51ce1a3b365971. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:13.717041 3674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c642583029ab46e49e51ce1a3b365971. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:13.719411 3947 tablet_service.cc:1467] Processing CreateTablet for tablet c642583029ab46e49e51ce1a3b365971 (DEFAULT_TABLE table=test-table [id=f786ac3c8a984b59ad8de9ac0d10b7b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:17:13.720901 3947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c642583029ab46e49e51ce1a3b365971. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:13.733381 4041 tablet_bootstrap.cc:492] T c642583029ab46e49e51ce1a3b365971 P b7a9401c551f4b2aa3bb0d32c39a4abd: Bootstrap starting.
I20260929 20:17:13.735510 4042 tablet_bootstrap.cc:492] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771: Bootstrap starting.
I20260929 20:17:13.737191 4043 tablet_bootstrap.cc:492] T c642583029ab46e49e51ce1a3b365971 P 5deeb3b1231942048cd724c8f314a9c1: Bootstrap starting.
I20260929 20:17:13.737970 4041 tablet_bootstrap.cc:654] T c642583029ab46e49e51ce1a3b365971 P b7a9401c551f4b2aa3bb0d32c39a4abd: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:13.739127 4042 tablet_bootstrap.cc:654] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:13.739347 4041 log.cc:824] T c642583029ab46e49e51ce1a3b365971 P b7a9401c551f4b2aa3bb0d32c39a4abd: Log is configured to *not* fsync() on all Append() calls
I20260929 20:17:13.740379 4043 tablet_bootstrap.cc:654] T c642583029ab46e49e51ce1a3b365971 P 5deeb3b1231942048cd724c8f314a9c1: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:13.740552 4042 log.cc:824] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771: Log is configured to *not* fsync() on all Append() calls
I20260929 20:17:13.741487 4043 log.cc:824] T c642583029ab46e49e51ce1a3b365971 P 5deeb3b1231942048cd724c8f314a9c1: Log is configured to *not* fsync() on all Append() calls
I20260929 20:17:13.743781 4042 tablet_bootstrap.cc:492] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771: No bootstrap required, opened a new log
I20260929 20:17:13.743916 4041 tablet_bootstrap.cc:492] T c642583029ab46e49e51ce1a3b365971 P b7a9401c551f4b2aa3bb0d32c39a4abd: No bootstrap required, opened a new log
I20260929 20:17:13.744098 4042 ts_tablet_manager.cc:1397] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 20:17:13.744204 4041 ts_tablet_manager.cc:1397] T c642583029ab46e49e51ce1a3b365971 P b7a9401c551f4b2aa3bb0d32c39a4abd: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260929 20:17:13.744602 4043 tablet_bootstrap.cc:492] T c642583029ab46e49e51ce1a3b365971 P 5deeb3b1231942048cd724c8f314a9c1: No bootstrap required, opened a new log
I20260929 20:17:13.744864 4043 ts_tablet_manager.cc:1397] T c642583029ab46e49e51ce1a3b365971 P 5deeb3b1231942048cd724c8f314a9c1: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260929 20:17:13.751672 4042 raft_consensus.cc:348] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:13.752070 4042 raft_consensus.cc:374] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:13.752175 4042 raft_consensus.cc:729] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57b2ac153bca44e9b3e4e5c82e3eb771, State: Initialized, Role: FOLLOWER
I20260929 20:17:13.752820 4042 consensus_queue.cc:260] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:13.754603 4042 ts_tablet_manager.cc:1428] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260929 20:17:13.754967 4041 raft_consensus.cc:348] T c642583029ab46e49e51ce1a3b365971 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:13.755446 4041 raft_consensus.cc:374] T c642583029ab46e49e51ce1a3b365971 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:13.755548 4041 raft_consensus.cc:729] T c642583029ab46e49e51ce1a3b365971 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7a9401c551f4b2aa3bb0d32c39a4abd, State: Initialized, Role: FOLLOWER
I20260929 20:17:13.755255 4043 raft_consensus.cc:348] T c642583029ab46e49e51ce1a3b365971 P 5deeb3b1231942048cd724c8f314a9c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:13.755694 4043 raft_consensus.cc:374] T c642583029ab46e49e51ce1a3b365971 P 5deeb3b1231942048cd724c8f314a9c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:13.755777 4043 raft_consensus.cc:729] T c642583029ab46e49e51ce1a3b365971 P 5deeb3b1231942048cd724c8f314a9c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5deeb3b1231942048cd724c8f314a9c1, State: Initialized, Role: FOLLOWER
I20260929 20:17:13.756153 4041 consensus_queue.cc:260] T c642583029ab46e49e51ce1a3b365971 P b7a9401c551f4b2aa3bb0d32c39a4abd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:13.756300 4043 consensus_queue.cc:260] T c642583029ab46e49e51ce1a3b365971 P 5deeb3b1231942048cd724c8f314a9c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:13.757905 3878 heartbeater.cc:503] Master 127.27.185.126:45075 was elected leader, sending a full tablet report...
I20260929 20:17:13.758152 4041 ts_tablet_manager.cc:1428] T c642583029ab46e49e51ce1a3b365971 P b7a9401c551f4b2aa3bb0d32c39a4abd: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 20:17:13.760989 4017 heartbeater.cc:503] Master 127.27.185.126:45075 was elected leader, sending a full tablet report...
I20260929 20:17:13.762108 4043 ts_tablet_manager.cc:1428] T c642583029ab46e49e51ce1a3b365971 P 5deeb3b1231942048cd724c8f314a9c1: Time spent starting tablet: real 0.017s user 0.015s sys 0.002s
W20260929 20:17:13.813333 3740 tablet.cc:2369] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:17:13.824352 4047 raft_consensus.cc:482] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:17:13.824714 4047 raft_consensus.cc:504] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:13.826381 4047 leader_election.cc:288] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7a9401c551f4b2aa3bb0d32c39a4abd (127.27.185.66:43357), 5deeb3b1231942048cd724c8f314a9c1 (127.27.185.67:33609)
I20260929 20:17:13.835870 4040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:13.826969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:52079 (local address 127.27.185.67:33609)
0929 20:17:13.827216 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:13.827223 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:13.827801 (+ 578us) server_negotiation.cc:407] Connection header received
0929 20:17:13.828012 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:13.828018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:13.828099 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:13.828202 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:17:13.829058 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.830216 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.833398 (+ 3182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.834639 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.834793 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:13.834869 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:13.835460 (+ 591us) server_negotiation.cc:299] Negotiation successful
0929 20:17:13.835624 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 20:17:13.836112 4050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:13.826828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:33609 (local address 127.27.185.65:52079)
0929 20:17:13.827685 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:13.827701 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:13.827857 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:13.828240 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:13.828251 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:13.828285 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:17:13.828871 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:13.828886 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.830360 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.830365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:13.833217 (+ 2852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:13.833234 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.834913 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.834922 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:13.835193 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:13.835303 (+ 110us) client_negotiation.cc:769] Sending connection context
0929 20:17:13.835436 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 20:17:13.835593 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"mutex_wait_us":293,"spinlock_wait_cycles":31360,"thread_start_us":395,"threads_started":1}
I20260929 20:17:13.836193 4038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:13.826846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:48443 (local address 127.27.185.66:43357)
0929 20:17:13.827050 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:13.827059 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:13.827800 (+ 741us) server_negotiation.cc:407] Connection header received
0929 20:17:13.828016 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:13.828021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:13.828099 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:13.828202 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:17:13.829058 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.830217 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.833402 (+ 3185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.834620 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.834802 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:13.834869 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:13.835735 (+ 866us) server_negotiation.cc:299] Negotiation successful
0929 20:17:13.835864 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:17:13.836568 4051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:13.826733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:43357 (local address 127.27.185.65:48443)
0929 20:17:13.827653 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:13.827677 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:13.827857 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:13.828240 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:13.828251 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:13.828287 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:17:13.828871 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:13.828886 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.830361 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.830365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:13.833243 (+ 2878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:13.833251 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.834930 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.834935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:13.835197 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:13.835304 (+ 107us) client_negotiation.cc:769] Sending connection context
0929 20:17:13.835436 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 20:17:13.835593 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"mutex_wait_us":319,"spinlock_wait_cycles":21760,"thread_start_us":688,"threads_started":1}
I20260929 20:17:13.837174 3967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c642583029ab46e49e51ce1a3b365971" candidate_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5deeb3b1231942048cd724c8f314a9c1" is_pre_election: true
I20260929 20:17:13.837543 3828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c642583029ab46e49e51ce1a3b365971" candidate_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" is_pre_election: true
I20260929 20:17:13.837813 3967 raft_consensus.cc:2454] T c642583029ab46e49e51ce1a3b365971 P 5deeb3b1231942048cd724c8f314a9c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57b2ac153bca44e9b3e4e5c82e3eb771 in term 0.
I20260929 20:17:13.838091 3828 raft_consensus.cc:2454] T c642583029ab46e49e51ce1a3b365971 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57b2ac153bca44e9b3e4e5c82e3eb771 in term 0.
I20260929 20:17:13.838598 3622 leader_election.cc:302] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57b2ac153bca44e9b3e4e5c82e3eb771, 5deeb3b1231942048cd724c8f314a9c1; no voters:
I20260929 20:17:13.838999 4047 raft_consensus.cc:2781] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:17:13.839150 4047 raft_consensus.cc:482] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:17:13.839263 4047 raft_consensus.cc:3037] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Advancing to term 1
W20260929 20:17:13.841609 3879 tablet.cc:2369] T c642583029ab46e49e51ce1a3b365971 P b7a9401c551f4b2aa3bb0d32c39a4abd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:17:13.841575 4047 raft_consensus.cc:504] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:13.842165 4047 leader_election.cc:288] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771 [CANDIDATE]: Term 1 election: Requested vote from peers b7a9401c551f4b2aa3bb0d32c39a4abd (127.27.185.66:43357), 5deeb3b1231942048cd724c8f314a9c1 (127.27.185.67:33609)
I20260929 20:17:13.842587 3828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c642583029ab46e49e51ce1a3b365971" candidate_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd"
I20260929 20:17:13.842669 3967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c642583029ab46e49e51ce1a3b365971" candidate_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5deeb3b1231942048cd724c8f314a9c1"
I20260929 20:17:13.842798 3828 raft_consensus.cc:3037] T c642583029ab46e49e51ce1a3b365971 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:17:13.842851 3967 raft_consensus.cc:3037] T c642583029ab46e49e51ce1a3b365971 P 5deeb3b1231942048cd724c8f314a9c1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:17:13.846078 3967 raft_consensus.cc:2454] T c642583029ab46e49e51ce1a3b365971 P 5deeb3b1231942048cd724c8f314a9c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57b2ac153bca44e9b3e4e5c82e3eb771 in term 1.
I20260929 20:17:13.846100 3828 raft_consensus.cc:2454] T c642583029ab46e49e51ce1a3b365971 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57b2ac153bca44e9b3e4e5c82e3eb771 in term 1.
I20260929 20:17:13.846570 3620 leader_election.cc:302] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 57b2ac153bca44e9b3e4e5c82e3eb771, 5deeb3b1231942048cd724c8f314a9c1, b7a9401c551f4b2aa3bb0d32c39a4abd; no voters:
I20260929 20:17:13.846903 4047 raft_consensus.cc:2781] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:17:13.847328 4047 raft_consensus.cc:686] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 1 LEADER]: Becoming Leader. State: Replica: 57b2ac153bca44e9b3e4e5c82e3eb771, State: Running, Role: LEADER
I20260929 20:17:13.848048 4047 consensus_queue.cc:237] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
W20260929 20:17:13.849591 4018 tablet.cc:2369] T c642583029ab46e49e51ce1a3b365971 P 5deeb3b1231942048cd724c8f314a9c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:17:13.852485 3531 catalog_manager.cc:5495] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771 reported cstate change: term changed from 0 to 1, leader changed from <none> to 57b2ac153bca44e9b3e4e5c82e3eb771 (127.27.185.65). New cstate: current_term: 1 leader_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } health_report { overall_health: UNKNOWN } } }
I20260929 20:17:13.876448 4037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:13.870434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38234 (local address 127.27.185.65:43463)
0929 20:17:13.870648 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:13.870654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:13.870676 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:17:13.870917 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:13.870925 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:13.871045 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:13.871206 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:17:13.871575 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.872657 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.874078 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:13.874705 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:13.874840 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:13.875237 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:17:13.875948 (+ 711us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:17:13.876059 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:13.876174 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 20:17:13.876244 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 20:17:13.884711 3967 raft_consensus.cc:1261] T c642583029ab46e49e51ce1a3b365971 P 5deeb3b1231942048cd724c8f314a9c1 [term 1 FOLLOWER]: Refusing update from remote peer 57b2ac153bca44e9b3e4e5c82e3eb771: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:17:13.884711 3828 raft_consensus.cc:1261] T c642583029ab46e49e51ce1a3b365971 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 1 FOLLOWER]: Refusing update from remote peer 57b2ac153bca44e9b3e4e5c82e3eb771: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:17:13.885717 4047 consensus_queue.cc:1035] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:17:13.886029 4052 consensus_queue.cc:1035] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:17:13.897864 4055 mvcc.cc:205] Tried to move back new op lower bound from 7334760586781184000 to 7334760586644959232. Current Snapshot: MvccSnapshot[applied={T|T < 7334760586781184000}]
I20260929 20:17:13.912825 3531 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:17:13.913087 3531 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:17:13.916424 3531 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 424e7e4d77534830b1f0ac96ba2e4982: Sending DeleteTablet for 3 replicas of tablet c642583029ab46e49e51ce1a3b365971
I20260929 20:17:13.917637 3674 tablet_service.cc:1514] Processing DeleteTablet for tablet c642583029ab46e49e51ce1a3b365971 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:17:13 UTC) from {username='oryx', principal='oryx/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:38230
I20260929 20:17:13.917726 3947 tablet_service.cc:1514] Processing DeleteTablet for tablet c642583029ab46e49e51ce1a3b365971 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:17:13 UTC) from {username='oryx', principal='oryx/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:44394
I20260929 20:17:13.918537 4066 tablet_replica.cc:330] T c642583029ab46e49e51ce1a3b365971 P 5deeb3b1231942048cd724c8f314a9c1: stopping tablet replica
I20260929 20:17:13.918584 3808 tablet_service.cc:1514] Processing DeleteTablet for tablet c642583029ab46e49e51ce1a3b365971 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:17:13 UTC) from {username='oryx', principal='oryx/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:42566
I20260929 20:17:13.919085 4066 raft_consensus.cc:2229] T c642583029ab46e49e51ce1a3b365971 P 5deeb3b1231942048cd724c8f314a9c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:17:13.919652 4066 raft_consensus.cc:2258] T c642583029ab46e49e51ce1a3b365971 P 5deeb3b1231942048cd724c8f314a9c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:17:13.919598 3530 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:17:13.920598 3530 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:17:13.920679 4065 tablet_replica.cc:330] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771: stopping tablet replica
I20260929 20:17:13.921448 4065 raft_consensus.cc:2229] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:17:13.922377 4067 tablet_replica.cc:330] T c642583029ab46e49e51ce1a3b365971 P b7a9401c551f4b2aa3bb0d32c39a4abd: stopping tablet replica
I20260929 20:17:13.922775 4067 raft_consensus.cc:2229] T c642583029ab46e49e51ce1a3b365971 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:17:13.923252 4067 raft_consensus.cc:2258] T c642583029ab46e49e51ce1a3b365971 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:17:13.923875 4065 raft_consensus.cc:2258] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:17:13.924099 4066 ts_tablet_manager.cc:1905] T c642583029ab46e49e51ce1a3b365971 P 5deeb3b1231942048cd724c8f314a9c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:17:13.925352 4067 ts_tablet_manager.cc:1905] T c642583029ab46e49e51ce1a3b365971 P b7a9401c551f4b2aa3bb0d32c39a4abd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:17:13.927096 4065 ts_tablet_manager.cc:1905] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:17:13.931097 3947 tablet_service.cc:1467] Processing CreateTablet for tablet 5df2d2faec7f45a3b53a60223f031ed9 (DEFAULT_TABLE table=test-table [id=710612f0aaa64baa97e49dd1b9ece317]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:17:13.931560 3947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5df2d2faec7f45a3b53a60223f031ed9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:13.931993 4067 ts_tablet_manager.cc:1918] T c642583029ab46e49e51ce1a3b365971 P b7a9401c551f4b2aa3bb0d32c39a4abd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:17:13.932153 4067 log.cc:1196] T c642583029ab46e49e51ce1a3b365971 P b7a9401c551f4b2aa3bb0d32c39a4abd: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-1/wal/wals/c642583029ab46e49e51ce1a3b365971
I20260929 20:17:13.932668 4067 ts_tablet_manager.cc:1939] T c642583029ab46e49e51ce1a3b365971 P b7a9401c551f4b2aa3bb0d32c39a4abd: Deleting consensus metadata
I20260929 20:17:13.934547 3517 catalog_manager.cc:4841] TS b7a9401c551f4b2aa3bb0d32c39a4abd (127.27.185.66:43357): tablet c642583029ab46e49e51ce1a3b365971 (table test-table [id=f786ac3c8a984b59ad8de9ac0d10b7b5]) successfully deleted
I20260929 20:17:13.934886 3808 tablet_service.cc:1467] Processing CreateTablet for tablet 5df2d2faec7f45a3b53a60223f031ed9 (DEFAULT_TABLE table=test-table [id=710612f0aaa64baa97e49dd1b9ece317]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:17:13.935314 3808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5df2d2faec7f45a3b53a60223f031ed9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:13.935992 3674 tablet_service.cc:1467] Processing CreateTablet for tablet 5df2d2faec7f45a3b53a60223f031ed9 (DEFAULT_TABLE table=test-table [id=710612f0aaa64baa97e49dd1b9ece317]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:17:13.937111 3674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5df2d2faec7f45a3b53a60223f031ed9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:13.938284 4066 ts_tablet_manager.cc:1918] T c642583029ab46e49e51ce1a3b365971 P 5deeb3b1231942048cd724c8f314a9c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:17:13.938531 4066 log.cc:1196] T c642583029ab46e49e51ce1a3b365971 P 5deeb3b1231942048cd724c8f314a9c1: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-2/wal/wals/c642583029ab46e49e51ce1a3b365971
I20260929 20:17:13.939134 4066 ts_tablet_manager.cc:1939] T c642583029ab46e49e51ce1a3b365971 P 5deeb3b1231942048cd724c8f314a9c1: Deleting consensus metadata
I20260929 20:17:13.940089 4043 tablet_bootstrap.cc:492] T 5df2d2faec7f45a3b53a60223f031ed9 P 5deeb3b1231942048cd724c8f314a9c1: Bootstrap starting.
I20260929 20:17:13.941030 3519 catalog_manager.cc:4841] TS 5deeb3b1231942048cd724c8f314a9c1 (127.27.185.67:33609): tablet c642583029ab46e49e51ce1a3b365971 (table test-table [id=f786ac3c8a984b59ad8de9ac0d10b7b5]) successfully deleted
I20260929 20:17:13.941690 4043 tablet_bootstrap.cc:654] T 5df2d2faec7f45a3b53a60223f031ed9 P 5deeb3b1231942048cd724c8f314a9c1: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:13.944175 4041 tablet_bootstrap.cc:492] T 5df2d2faec7f45a3b53a60223f031ed9 P b7a9401c551f4b2aa3bb0d32c39a4abd: Bootstrap starting.
I20260929 20:17:13.944658 4042 tablet_bootstrap.cc:492] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771: Bootstrap starting.
I20260929 20:17:13.945493 4043 tablet_bootstrap.cc:492] T 5df2d2faec7f45a3b53a60223f031ed9 P 5deeb3b1231942048cd724c8f314a9c1: No bootstrap required, opened a new log
I20260929 20:17:13.945621 4043 ts_tablet_manager.cc:1397] T 5df2d2faec7f45a3b53a60223f031ed9 P 5deeb3b1231942048cd724c8f314a9c1: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20260929 20:17:13.945787 4041 tablet_bootstrap.cc:654] T 5df2d2faec7f45a3b53a60223f031ed9 P b7a9401c551f4b2aa3bb0d32c39a4abd: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:13.946336 4042 tablet_bootstrap.cc:654] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:13.946382 4043 raft_consensus.cc:348] T 5df2d2faec7f45a3b53a60223f031ed9 P 5deeb3b1231942048cd724c8f314a9c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:13.946592 4043 raft_consensus.cc:374] T 5df2d2faec7f45a3b53a60223f031ed9 P 5deeb3b1231942048cd724c8f314a9c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:13.946691 4043 raft_consensus.cc:729] T 5df2d2faec7f45a3b53a60223f031ed9 P 5deeb3b1231942048cd724c8f314a9c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5deeb3b1231942048cd724c8f314a9c1, State: Initialized, Role: FOLLOWER
I20260929 20:17:13.946918 4043 consensus_queue.cc:260] T 5df2d2faec7f45a3b53a60223f031ed9 P 5deeb3b1231942048cd724c8f314a9c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:13.947499 4043 ts_tablet_manager.cc:1428] T 5df2d2faec7f45a3b53a60223f031ed9 P 5deeb3b1231942048cd724c8f314a9c1: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 20:17:13.949958 4065 ts_tablet_manager.cc:1918] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:17:13.950140 4065 log.cc:1196] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-0/wal/wals/c642583029ab46e49e51ce1a3b365971
I20260929 20:17:13.950698 4065 ts_tablet_manager.cc:1939] T c642583029ab46e49e51ce1a3b365971 P 57b2ac153bca44e9b3e4e5c82e3eb771: Deleting consensus metadata
I20260929 20:17:13.952373 3516 catalog_manager.cc:4841] TS 57b2ac153bca44e9b3e4e5c82e3eb771 (127.27.185.65:43463): tablet c642583029ab46e49e51ce1a3b365971 (table test-table [id=f786ac3c8a984b59ad8de9ac0d10b7b5]) successfully deleted
I20260929 20:17:13.957216 4042 tablet_bootstrap.cc:492] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771: No bootstrap required, opened a new log
I20260929 20:17:13.957412 4042 ts_tablet_manager.cc:1397] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20260929 20:17:13.958521 4042 raft_consensus.cc:348] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:13.958834 4042 raft_consensus.cc:374] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:13.958953 4042 raft_consensus.cc:729] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57b2ac153bca44e9b3e4e5c82e3eb771, State: Initialized, Role: FOLLOWER
I20260929 20:17:13.959357 4042 consensus_queue.cc:260] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:13.961165 4042 ts_tablet_manager.cc:1428] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260929 20:17:13.963387 4041 tablet_bootstrap.cc:492] T 5df2d2faec7f45a3b53a60223f031ed9 P b7a9401c551f4b2aa3bb0d32c39a4abd: No bootstrap required, opened a new log
I20260929 20:17:13.963635 4041 ts_tablet_manager.cc:1397] T 5df2d2faec7f45a3b53a60223f031ed9 P b7a9401c551f4b2aa3bb0d32c39a4abd: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.009s
I20260929 20:17:13.964424 4041 raft_consensus.cc:348] T 5df2d2faec7f45a3b53a60223f031ed9 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:13.964628 4041 raft_consensus.cc:374] T 5df2d2faec7f45a3b53a60223f031ed9 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:13.964720 4041 raft_consensus.cc:729] T 5df2d2faec7f45a3b53a60223f031ed9 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7a9401c551f4b2aa3bb0d32c39a4abd, State: Initialized, Role: FOLLOWER
I20260929 20:17:13.965026 4041 consensus_queue.cc:260] T 5df2d2faec7f45a3b53a60223f031ed9 P b7a9401c551f4b2aa3bb0d32c39a4abd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:13.965587 4041 ts_tablet_manager.cc:1428] T 5df2d2faec7f45a3b53a60223f031ed9 P b7a9401c551f4b2aa3bb0d32c39a4abd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 20:17:14.139034 4047 raft_consensus.cc:482] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:17:14.139215 4047 raft_consensus.cc:504] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:14.139961 4047 leader_election.cc:288] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7a9401c551f4b2aa3bb0d32c39a4abd (127.27.185.66:43357), 5deeb3b1231942048cd724c8f314a9c1 (127.27.185.67:33609)
I20260929 20:17:14.140522 3967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df2d2faec7f45a3b53a60223f031ed9" candidate_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5deeb3b1231942048cd724c8f314a9c1" is_pre_election: true
I20260929 20:17:14.140550 3828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df2d2faec7f45a3b53a60223f031ed9" candidate_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" is_pre_election: true
I20260929 20:17:14.140774 3967 raft_consensus.cc:2454] T 5df2d2faec7f45a3b53a60223f031ed9 P 5deeb3b1231942048cd724c8f314a9c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57b2ac153bca44e9b3e4e5c82e3eb771 in term 0.
I20260929 20:17:14.140833 3828 raft_consensus.cc:2454] T 5df2d2faec7f45a3b53a60223f031ed9 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57b2ac153bca44e9b3e4e5c82e3eb771 in term 0.
I20260929 20:17:14.141227 3622 leader_election.cc:302] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57b2ac153bca44e9b3e4e5c82e3eb771, 5deeb3b1231942048cd724c8f314a9c1; no voters:
I20260929 20:17:14.141506 4047 raft_consensus.cc:2781] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:17:14.141657 4047 raft_consensus.cc:482] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:17:14.141793 4047 raft_consensus.cc:3037] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:17:14.143993 4047 raft_consensus.cc:504] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:14.144629 4047 leader_election.cc:288] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771 [CANDIDATE]: Term 1 election: Requested vote from peers b7a9401c551f4b2aa3bb0d32c39a4abd (127.27.185.66:43357), 5deeb3b1231942048cd724c8f314a9c1 (127.27.185.67:33609)
I20260929 20:17:14.145114 3828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df2d2faec7f45a3b53a60223f031ed9" candidate_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd"
I20260929 20:17:14.145198 3967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df2d2faec7f45a3b53a60223f031ed9" candidate_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5deeb3b1231942048cd724c8f314a9c1"
I20260929 20:17:14.145336 3828 raft_consensus.cc:3037] T 5df2d2faec7f45a3b53a60223f031ed9 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:17:14.145380 3967 raft_consensus.cc:3037] T 5df2d2faec7f45a3b53a60223f031ed9 P 5deeb3b1231942048cd724c8f314a9c1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:17:14.148249 3828 raft_consensus.cc:2454] T 5df2d2faec7f45a3b53a60223f031ed9 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57b2ac153bca44e9b3e4e5c82e3eb771 in term 1.
I20260929 20:17:14.148249 3967 raft_consensus.cc:2454] T 5df2d2faec7f45a3b53a60223f031ed9 P 5deeb3b1231942048cd724c8f314a9c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57b2ac153bca44e9b3e4e5c82e3eb771 in term 1.
I20260929 20:17:14.148638 3620 leader_election.cc:302] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57b2ac153bca44e9b3e4e5c82e3eb771, b7a9401c551f4b2aa3bb0d32c39a4abd; no voters:
I20260929 20:17:14.148947 4047 raft_consensus.cc:2781] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:17:14.149139 4047 raft_consensus.cc:686] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 1 LEADER]: Becoming Leader. State: Replica: 57b2ac153bca44e9b3e4e5c82e3eb771, State: Running, Role: LEADER
I20260929 20:17:14.149463 4047 consensus_queue.cc:237] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:14.152256 3530 catalog_manager.cc:5495] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771 reported cstate change: term changed from 0 to 1, leader changed from <none> to 57b2ac153bca44e9b3e4e5c82e3eb771 (127.27.185.65). New cstate: current_term: 1 leader_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } health_report { overall_health: UNKNOWN } } }
Sep 29 20:17:14 dist-test-slave-4cx3 krb5kdc[3487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713031, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.126@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM
I20260929 20:17:14.216851 4078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:14.202079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:45075 (local address 127.0.0.1:59324)
0929 20:17:14.202901 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:14.202916 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:14.203113 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:14.203498 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:14.203505 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:14.204263 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:14.204546 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:14.204556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.205797 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.205805 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:14.206846 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:14.206858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.207623 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.207630 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:14.207812 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:14.208530 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:14.208547 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:14.210905 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:14.214208 (+ 3303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:14.214223 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:14.214231 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:14.214629 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:14.215009 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:14.215013 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:14.215016 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:14.215137 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:14.215740 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:14.215748 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:14.215913 (+ 165us) client_negotiation.cc:769] Sending connection context
0929 20:17:14.216130 (+ 217us) client_negotiation.cc:240] Negotiation successful
0929 20:17:14.216409 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":535,"threads_started":1}
I20260929 20:17:14.217344 4077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:14.202015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59324 (local address 127.27.185.126:45075)
0929 20:17:14.202724 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:14.202732 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:14.203019 (+ 287us) server_negotiation.cc:407] Connection header received
0929 20:17:14.203234 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:14.203240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:14.203297 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:14.203459 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:14.204701 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.205678 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.206992 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.207375 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.207563 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:14.211150 (+ 3587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:14.211154 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:14.211157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:14.211184 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:14.214080 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:14.214746 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:14.214753 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:14.214758 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:14.214887 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:14.215252 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:14.215256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:14.215260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:14.215593 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:14.215747 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:14.216280 (+ 533us) server_negotiation.cc:299] Negotiation successful
0929 20:17:14.216579 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":509,"thread_start_us":321,"threads_started":1}
I20260929 20:17:14.222677 3530 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:59324:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 20:17:14.224047 3530 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:17:14.230965 3674 tablet_service.cc:1467] Processing CreateTablet for tablet f51dd3ae84144d97924c92660c182362 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=59bab6de85f5451886ceb99f3756b952]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:17:14.231495 3947 tablet_service.cc:1467] Processing CreateTablet for tablet f51dd3ae84144d97924c92660c182362 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=59bab6de85f5451886ceb99f3756b952]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:17:14.231755 3674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f51dd3ae84144d97924c92660c182362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:14.231541 3808 tablet_service.cc:1467] Processing CreateTablet for tablet f51dd3ae84144d97924c92660c182362 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=59bab6de85f5451886ceb99f3756b952]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:17:14.232062 3947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f51dd3ae84144d97924c92660c182362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:14.232138 3808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f51dd3ae84144d97924c92660c182362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:14.239498 4043 tablet_bootstrap.cc:492] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1: Bootstrap starting.
I20260929 20:17:14.239601 4042 tablet_bootstrap.cc:492] T f51dd3ae84144d97924c92660c182362 P 57b2ac153bca44e9b3e4e5c82e3eb771: Bootstrap starting.
I20260929 20:17:14.240978 4042 tablet_bootstrap.cc:654] T f51dd3ae84144d97924c92660c182362 P 57b2ac153bca44e9b3e4e5c82e3eb771: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:14.241269 4043 tablet_bootstrap.cc:654] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:14.243206 4042 tablet_bootstrap.cc:492] T f51dd3ae84144d97924c92660c182362 P 57b2ac153bca44e9b3e4e5c82e3eb771: No bootstrap required, opened a new log
I20260929 20:17:14.243424 4042 ts_tablet_manager.cc:1397] T f51dd3ae84144d97924c92660c182362 P 57b2ac153bca44e9b3e4e5c82e3eb771: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 20:17:14.244335 4041 tablet_bootstrap.cc:492] T f51dd3ae84144d97924c92660c182362 P b7a9401c551f4b2aa3bb0d32c39a4abd: Bootstrap starting.
I20260929 20:17:14.245867 4041 tablet_bootstrap.cc:654] T f51dd3ae84144d97924c92660c182362 P b7a9401c551f4b2aa3bb0d32c39a4abd: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:14.245988 4043 tablet_bootstrap.cc:492] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1: No bootstrap required, opened a new log
I20260929 20:17:14.246166 4043 ts_tablet_manager.cc:1397] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260929 20:17:14.246809 4042 raft_consensus.cc:348] T f51dd3ae84144d97924c92660c182362 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:14.246934 4043 raft_consensus.cc:348] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:14.247085 4042 raft_consensus.cc:374] T f51dd3ae84144d97924c92660c182362 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:14.247172 4042 raft_consensus.cc:729] T f51dd3ae84144d97924c92660c182362 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57b2ac153bca44e9b3e4e5c82e3eb771, State: Initialized, Role: FOLLOWER
I20260929 20:17:14.247184 4043 raft_consensus.cc:374] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:14.247257 4043 raft_consensus.cc:729] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5deeb3b1231942048cd724c8f314a9c1, State: Initialized, Role: FOLLOWER
I20260929 20:17:14.247479 4042 consensus_queue.cc:260] T f51dd3ae84144d97924c92660c182362 P 57b2ac153bca44e9b3e4e5c82e3eb771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:14.247525 4043 consensus_queue.cc:260] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:14.247905 4047 tablet_replica.cc:439] T f51dd3ae84144d97924c92660c182362 P 57b2ac153bca44e9b3e4e5c82e3eb771: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } } }
I20260929 20:17:14.247982 4049 tablet_replica.cc:439] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } } }
I20260929 20:17:14.248260 4049 tablet_replica.cc:442] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:17:14.248260 4047 tablet_replica.cc:442] T f51dd3ae84144d97924c92660c182362 P 57b2ac153bca44e9b3e4e5c82e3eb771: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:17:14.248850 4043 ts_tablet_manager.cc:1428] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 20:17:14.249704 4042 ts_tablet_manager.cc:1428] T f51dd3ae84144d97924c92660c182362 P 57b2ac153bca44e9b3e4e5c82e3eb771: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260929 20:17:14.252204 4041 tablet_bootstrap.cc:492] T f51dd3ae84144d97924c92660c182362 P b7a9401c551f4b2aa3bb0d32c39a4abd: No bootstrap required, opened a new log
I20260929 20:17:14.252403 4041 ts_tablet_manager.cc:1397] T f51dd3ae84144d97924c92660c182362 P b7a9401c551f4b2aa3bb0d32c39a4abd: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20260929 20:17:14.253069 4041 raft_consensus.cc:348] T f51dd3ae84144d97924c92660c182362 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:14.253288 4041 raft_consensus.cc:374] T f51dd3ae84144d97924c92660c182362 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:14.253392 4041 raft_consensus.cc:729] T f51dd3ae84144d97924c92660c182362 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7a9401c551f4b2aa3bb0d32c39a4abd, State: Initialized, Role: FOLLOWER
I20260929 20:17:14.253659 4041 consensus_queue.cc:260] T f51dd3ae84144d97924c92660c182362 P b7a9401c551f4b2aa3bb0d32c39a4abd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:14.254115 4048 tablet_replica.cc:439] T f51dd3ae84144d97924c92660c182362 P b7a9401c551f4b2aa3bb0d32c39a4abd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } } }
I20260929 20:17:14.254384 4048 tablet_replica.cc:442] T f51dd3ae84144d97924c92660c182362 P b7a9401c551f4b2aa3bb0d32c39a4abd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:17:14.255230 4041 ts_tablet_manager.cc:1428] T f51dd3ae84144d97924c92660c182362 P b7a9401c551f4b2aa3bb0d32c39a4abd: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260929 20:17:14.311172 4049 raft_consensus.cc:482] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:17:14.311460 4049 raft_consensus.cc:504] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:14.313117 4049 leader_election.cc:288] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57b2ac153bca44e9b3e4e5c82e3eb771 (127.27.185.65:43463), b7a9401c551f4b2aa3bb0d32c39a4abd (127.27.185.66:43357)
I20260929 20:17:14.322562 4038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:14.313844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:54135 (local address 127.27.185.66:43357)
0929 20:17:14.314023 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:14.314029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:14.315873 (+ 1844us) server_negotiation.cc:407] Connection header received
0929 20:17:14.316028 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:14.316035 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:14.316137 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:14.316226 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:17:14.316818 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.317711 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.320429 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.321635 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.321807 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:14.321904 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:14.322118 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 20:17:14.322267 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:17:14.322605 4082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:14.313401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:43357 (local address 127.27.185.67:54135)
0929 20:17:14.315439 (+ 2038us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:14.315456 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:14.315904 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:14.316259 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:14.316264 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:14.316289 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:17:14.316647 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:14.316658 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.317877 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.317884 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:14.320257 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:14.320272 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.321796 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.321800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:14.321939 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:14.321963 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 20:17:14.322087 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 20:17:14.322217 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1905,"mutex_wait_us":129,"thread_start_us":1766,"threads_started":1}
I20260929 20:17:14.323705 3828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f51dd3ae84144d97924c92660c182362" candidate_uuid: "5deeb3b1231942048cd724c8f314a9c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" is_pre_election: true
I20260929 20:17:14.324035 3828 raft_consensus.cc:2454] T f51dd3ae84144d97924c92660c182362 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5deeb3b1231942048cd724c8f314a9c1 in term 0.
I20260929 20:17:14.324512 4083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:14.313365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:43463 (local address 127.27.185.67:54803)
0929 20:17:14.314960 (+ 1595us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:14.314982 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:14.315619 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:14.316049 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:14.316053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:14.316076 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:17:14.316360 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:14.316377 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.317842 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.317849 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:14.320257 (+ 2408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:14.320272 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.323615 (+ 3343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.323621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:14.323797 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:14.323830 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:17:14.324003 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 20:17:14.324139 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":943,"mutex_wait_us":117,"thread_start_us":799,"threads_started":1}
I20260929 20:17:14.324550 4037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:14.313731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:54803 (local address 127.27.185.65:43463)
0929 20:17:14.313972 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:14.313982 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:14.315069 (+ 1087us) server_negotiation.cc:407] Connection header received
0929 20:17:14.315854 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:14.315859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:14.315935 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:14.316023 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:17:14.316518 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.317695 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.321990 (+ 4295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.323343 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.323579 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:14.323672 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:14.324050 (+ 378us) server_negotiation.cc:299] Negotiation successful
0929 20:17:14.324209 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 20:17:14.324664 3899 leader_election.cc:302] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5deeb3b1231942048cd724c8f314a9c1, b7a9401c551f4b2aa3bb0d32c39a4abd; no voters:
I20260929 20:17:14.325050 4049 raft_consensus.cc:2781] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:17:14.325172 4049 raft_consensus.cc:482] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:17:14.325278 4049 raft_consensus.cc:3037] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:17:14.325760 3694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f51dd3ae84144d97924c92660c182362" candidate_uuid: "5deeb3b1231942048cd724c8f314a9c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" is_pre_election: true
I20260929 20:17:14.326184 3694 raft_consensus.cc:2454] T f51dd3ae84144d97924c92660c182362 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5deeb3b1231942048cd724c8f314a9c1 in term 0.
I20260929 20:17:14.327766 4049 raft_consensus.cc:504] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:14.328397 4049 leader_election.cc:288] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1 [CANDIDATE]: Term 1 election: Requested vote from peers 57b2ac153bca44e9b3e4e5c82e3eb771 (127.27.185.65:43463), b7a9401c551f4b2aa3bb0d32c39a4abd (127.27.185.66:43357)
I20260929 20:17:14.328855 3694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f51dd3ae84144d97924c92660c182362" candidate_uuid: "5deeb3b1231942048cd724c8f314a9c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771"
I20260929 20:17:14.328943 3828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f51dd3ae84144d97924c92660c182362" candidate_uuid: "5deeb3b1231942048cd724c8f314a9c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd"
I20260929 20:17:14.329073 3694 raft_consensus.cc:3037] T f51dd3ae84144d97924c92660c182362 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:17:14.329195 3828 raft_consensus.cc:3037] T f51dd3ae84144d97924c92660c182362 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:17:14.331871 3694 raft_consensus.cc:2454] T f51dd3ae84144d97924c92660c182362 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5deeb3b1231942048cd724c8f314a9c1 in term 1.
I20260929 20:17:14.332267 3828 raft_consensus.cc:2454] T f51dd3ae84144d97924c92660c182362 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5deeb3b1231942048cd724c8f314a9c1 in term 1.
I20260929 20:17:14.332291 3898 leader_election.cc:302] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57b2ac153bca44e9b3e4e5c82e3eb771, 5deeb3b1231942048cd724c8f314a9c1; no voters:
I20260929 20:17:14.332653 4049 raft_consensus.cc:2781] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:17:14.333102 4049 raft_consensus.cc:686] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1 [term 1 LEADER]: Becoming Leader. State: Replica: 5deeb3b1231942048cd724c8f314a9c1, State: Running, Role: LEADER
I20260929 20:17:14.333515 4049 consensus_queue.cc:237] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:14.335951 4084 tablet_replica.cc:439] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1: TxnStatusTablet state changed. Reason: New leader 5deeb3b1231942048cd724c8f314a9c1. Latest consensus state: current_term: 1 leader_uuid: "5deeb3b1231942048cd724c8f314a9c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } } }
I20260929 20:17:14.336380 4084 tablet_replica.cc:442] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:17:14.336881 4086 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 20:17:14.337301 4086 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 20:17:14.337467 3531 catalog_manager.cc:5495] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5deeb3b1231942048cd724c8f314a9c1 (127.27.185.67). New cstate: current_term: 1 leader_uuid: "5deeb3b1231942048cd724c8f314a9c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } health_report { overall_health: HEALTHY } } }
I20260929 20:17:14.366356 4078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:14.358634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:33609 (local address 127.0.0.1:44408)
0929 20:17:14.359041 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:14.359071 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:14.359261 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:14.360410 (+ 1149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:14.360415 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:14.360428 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:17:14.360650 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:14.360665 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.362113 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.362119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:14.363500 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:14.363519 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.364427 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.364433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:14.364636 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:14.364695 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:17:14.365763 (+ 1068us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:17:14.365785 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 20:17:14.365871 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 20:17:14.366048 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194}
I20260929 20:17:14.366688 4089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:14.358695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44408 (local address 127.27.185.67:33609)
0929 20:17:14.360006 (+ 1311us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:14.360011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:14.360032 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:17:14.360117 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:14.360120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:14.360206 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:14.360385 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:17:14.360834 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.361856 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.363704 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.364240 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.364385 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:14.364979 (+ 594us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:17:14.365646 (+ 667us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:17:14.365806 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:14.365930 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 20:17:14.366030 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1178,"thread_start_us":1057,"threads_started":1}
I20260929 20:17:14.370980 3828 raft_consensus.cc:1261] T f51dd3ae84144d97924c92660c182362 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 1 FOLLOWER]: Refusing update from remote peer 5deeb3b1231942048cd724c8f314a9c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:17:14.371423 3694 raft_consensus.cc:1261] T f51dd3ae84144d97924c92660c182362 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 1 FOLLOWER]: Refusing update from remote peer 5deeb3b1231942048cd724c8f314a9c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:17:14.372179 4049 consensus_queue.cc:1035] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:17:14.372638 4084 consensus_queue.cc:1035] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:17:14.383607 4047 tablet_replica.cc:439] T f51dd3ae84144d97924c92660c182362 P 57b2ac153bca44e9b3e4e5c82e3eb771: TxnStatusTablet state changed. Reason: New leader 5deeb3b1231942048cd724c8f314a9c1. Latest consensus state: current_term: 1 leader_uuid: "5deeb3b1231942048cd724c8f314a9c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } } }
I20260929 20:17:14.383908 4047 tablet_replica.cc:442] T f51dd3ae84144d97924c92660c182362 P 57b2ac153bca44e9b3e4e5c82e3eb771: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:17:14.387260 4048 tablet_replica.cc:439] T f51dd3ae84144d97924c92660c182362 P b7a9401c551f4b2aa3bb0d32c39a4abd: TxnStatusTablet state changed. Reason: New leader 5deeb3b1231942048cd724c8f314a9c1. Latest consensus state: current_term: 1 leader_uuid: "5deeb3b1231942048cd724c8f314a9c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } } }
I20260929 20:17:14.387639 4048 tablet_replica.cc:442] T f51dd3ae84144d97924c92660c182362 P b7a9401c551f4b2aa3bb0d32c39a4abd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:17:14.389269 4084 tablet_replica.cc:439] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5deeb3b1231942048cd724c8f314a9c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } } }
I20260929 20:17:14.389549 4084 tablet_replica.cc:442] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:17:14.392047 4047 tablet_replica.cc:439] T f51dd3ae84144d97924c92660c182362 P 57b2ac153bca44e9b3e4e5c82e3eb771: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5deeb3b1231942048cd724c8f314a9c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } } }
I20260929 20:17:14.392319 4047 tablet_replica.cc:442] T f51dd3ae84144d97924c92660c182362 P 57b2ac153bca44e9b3e4e5c82e3eb771: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:17:14.396113 4084 tablet_replica.cc:439] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5deeb3b1231942048cd724c8f314a9c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } } }
I20260929 20:17:14.396415 4084 tablet_replica.cc:442] T f51dd3ae84144d97924c92660c182362 P 5deeb3b1231942048cd724c8f314a9c1: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:17:14.399605 4048 tablet_replica.cc:439] T f51dd3ae84144d97924c92660c182362 P b7a9401c551f4b2aa3bb0d32c39a4abd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5deeb3b1231942048cd724c8f314a9c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } } }
I20260929 20:17:14.399849 4048 tablet_replica.cc:442] T f51dd3ae84144d97924c92660c182362 P b7a9401c551f4b2aa3bb0d32c39a4abd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:17:14.414592 4095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:14.408640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:33609 (local address 127.27.185.65:34797)
0929 20:17:14.409324 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:14.409341 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:14.409453 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:14.409795 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:14.409798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:14.409807 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:17:14.410021 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:14.410028 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.411518 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.411521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:14.412647 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:14.412659 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.413586 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.413590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:14.413702 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:14.413752 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:17:14.414148 (+ 396us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:17:14.414155 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 20:17:14.414214 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 20:17:14.414310 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":346,"threads_started":1}
I20260929 20:17:14.414616 4089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:14.408696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:34797 (local address 127.27.185.67:33609)
0929 20:17:14.408891 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:14.408898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:14.409424 (+ 526us) server_negotiation.cc:407] Connection header received
0929 20:17:14.409586 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:14.409592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:14.409676 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:14.409786 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:17:14.410158 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.411312 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.412776 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.413433 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.413564 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:14.413918 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:17:14.414045 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:17:14.414142 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:14.414243 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 20:17:14.414318 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:17:14.424630 3828 raft_consensus.cc:1261] T 5df2d2faec7f45a3b53a60223f031ed9 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 1 FOLLOWER]: Refusing update from remote peer 57b2ac153bca44e9b3e4e5c82e3eb771: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:17:14.424690 3967 raft_consensus.cc:1261] T 5df2d2faec7f45a3b53a60223f031ed9 P 5deeb3b1231942048cd724c8f314a9c1 [term 1 FOLLOWER]: Refusing update from remote peer 57b2ac153bca44e9b3e4e5c82e3eb771: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:17:14.425165 4047 consensus_queue.cc:1035] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:17:14.425486 4052 consensus_queue.cc:1035] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:17:14.435344 4057 mvcc.cc:205] Tried to move back new op lower bound from 7334760588998434816 to 7334760587878506496. Current Snapshot: MvccSnapshot[applied={T|T < 7334760588998434816}]
I20260929 20:17:14.460707 4037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:14.454743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:41285 (local address 127.27.185.65:43463)
0929 20:17:14.454912 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:14.454918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:14.455589 (+ 671us) server_negotiation.cc:407] Connection header received
0929 20:17:14.455727 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:14.455732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:14.455814 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:14.455913 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:17:14.456554 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.457324 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.458819 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.459256 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.459400 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:14.459976 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:17:14.460135 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:17:14.460221 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:14.460337 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 20:17:14.460417 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 20:17:14.461191 4100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:14.454834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:43463 (local address 127.27.185.67:41285)
0929 20:17:14.455489 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:14.455507 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:14.455625 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:14.455927 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:14.455930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:14.455938 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:17:14.456326 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:14.456339 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.457481 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.457498 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:14.458660 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:14.458673 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.459469 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.459479 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:14.459667 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:14.459801 (+ 134us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:17:14.460222 (+ 421us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:17:14.460236 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 20:17:14.460318 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 20:17:14.460464 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":303,"threads_started":1}
I20260929 20:17:14.507225 3531 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:17:14.507514 3531 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:17:14.511220 3531 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 424e7e4d77534830b1f0ac96ba2e4982: Sending DeleteTablet for 3 replicas of tablet 5df2d2faec7f45a3b53a60223f031ed9
I20260929 20:17:14.512327 3674 tablet_service.cc:1514] Processing DeleteTablet for tablet 5df2d2faec7f45a3b53a60223f031ed9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:17:14 UTC) from {username='oryx', principal='oryx/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:38230
I20260929 20:17:14.512759 3808 tablet_service.cc:1514] Processing DeleteTablet for tablet 5df2d2faec7f45a3b53a60223f031ed9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:17:14 UTC) from {username='oryx', principal='oryx/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:42566
I20260929 20:17:14.518651 3947 tablet_service.cc:1514] Processing DeleteTablet for tablet 5df2d2faec7f45a3b53a60223f031ed9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:17:14 UTC) from {username='oryx', principal='oryx/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:44394
I20260929 20:17:14.525136 4103 tablet_replica.cc:330] T 5df2d2faec7f45a3b53a60223f031ed9 P b7a9401c551f4b2aa3bb0d32c39a4abd: stopping tablet replica
I20260929 20:17:14.525641 4103 raft_consensus.cc:2229] T 5df2d2faec7f45a3b53a60223f031ed9 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:17:14.525964 4103 raft_consensus.cc:2258] T 5df2d2faec7f45a3b53a60223f031ed9 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:17:14.527423 4103 ts_tablet_manager.cc:1905] T 5df2d2faec7f45a3b53a60223f031ed9 P b7a9401c551f4b2aa3bb0d32c39a4abd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:17:14.529757 4104 tablet_replica.cc:330] T 5df2d2faec7f45a3b53a60223f031ed9 P 5deeb3b1231942048cd724c8f314a9c1: stopping tablet replica
I20260929 20:17:14.529762 4102 tablet_replica.cc:330] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771: stopping tablet replica
I20260929 20:17:14.529960 4102 raft_consensus.cc:2229] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:17:14.530171 4104 raft_consensus.cc:2229] T 5df2d2faec7f45a3b53a60223f031ed9 P 5deeb3b1231942048cd724c8f314a9c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:17:14.530376 4102 raft_consensus.cc:2258] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:17:14.530561 4104 raft_consensus.cc:2258] T 5df2d2faec7f45a3b53a60223f031ed9 P 5deeb3b1231942048cd724c8f314a9c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:17:14.531545 4102 ts_tablet_manager.cc:1905] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:17:14.531705 4104 ts_tablet_manager.cc:1905] T 5df2d2faec7f45a3b53a60223f031ed9 P 5deeb3b1231942048cd724c8f314a9c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:17:14.539458 4102 ts_tablet_manager.cc:1918] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:17:14.539474 4103 ts_tablet_manager.cc:1918] T 5df2d2faec7f45a3b53a60223f031ed9 P b7a9401c551f4b2aa3bb0d32c39a4abd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:17:14.539485 4104 ts_tablet_manager.cc:1918] T 5df2d2faec7f45a3b53a60223f031ed9 P 5deeb3b1231942048cd724c8f314a9c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:17:14.539616 4103 log.cc:1196] T 5df2d2faec7f45a3b53a60223f031ed9 P b7a9401c551f4b2aa3bb0d32c39a4abd: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-1/wal/wals/5df2d2faec7f45a3b53a60223f031ed9
I20260929 20:17:14.539616 4102 log.cc:1196] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-0/wal/wals/5df2d2faec7f45a3b53a60223f031ed9
I20260929 20:17:14.539813 4104 log.cc:1196] T 5df2d2faec7f45a3b53a60223f031ed9 P 5deeb3b1231942048cd724c8f314a9c1: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-2/wal/wals/5df2d2faec7f45a3b53a60223f031ed9
I20260929 20:17:14.540087 4102 ts_tablet_manager.cc:1939] T 5df2d2faec7f45a3b53a60223f031ed9 P 57b2ac153bca44e9b3e4e5c82e3eb771: Deleting consensus metadata
I20260929 20:17:14.540100 4103 ts_tablet_manager.cc:1939] T 5df2d2faec7f45a3b53a60223f031ed9 P b7a9401c551f4b2aa3bb0d32c39a4abd: Deleting consensus metadata
I20260929 20:17:14.540416 4104 ts_tablet_manager.cc:1939] T 5df2d2faec7f45a3b53a60223f031ed9 P 5deeb3b1231942048cd724c8f314a9c1: Deleting consensus metadata
I20260929 20:17:14.541236 3516 catalog_manager.cc:4841] TS 57b2ac153bca44e9b3e4e5c82e3eb771 (127.27.185.65:43463): tablet 5df2d2faec7f45a3b53a60223f031ed9 (table test-table [id=710612f0aaa64baa97e49dd1b9ece317]) successfully deleted
I20260929 20:17:14.541597 3519 catalog_manager.cc:4841] TS 5deeb3b1231942048cd724c8f314a9c1 (127.27.185.67:33609): tablet 5df2d2faec7f45a3b53a60223f031ed9 (table test-table [id=710612f0aaa64baa97e49dd1b9ece317]) successfully deleted
I20260929 20:17:14.541599 3517 catalog_manager.cc:4841] TS b7a9401c551f4b2aa3bb0d32c39a4abd (127.27.185.66:43357): tablet 5df2d2faec7f45a3b53a60223f031ed9 (table test-table [id=710612f0aaa64baa97e49dd1b9ece317]) successfully deleted
Sep 29 20:17:14 dist-test-slave-4cx3 krb5kdc[3487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 20:17:14 dist-test-slave-4cx3 krb5kdc[3487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:14 dist-test-slave-4cx3 krb5kdc[3487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 20:17:14.586239 4077 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:14.574735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59338 (local address 127.27.185.126:45075)
0929 20:17:14.574974 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:14.574980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:14.575430 (+ 450us) server_negotiation.cc:407] Connection header received
0929 20:17:14.575676 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:14.575682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:14.575756 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:14.575880 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:14.577441 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.578576 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.580123 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.580569 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.580713 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:14.585961 (+ 5248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59338: BlockingRecv error: recv got EOF from 127.0.0.1:59338 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 20:17:14 dist-test-slave-4cx3 krb5kdc[3487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM
I20260929 20:17:14.607577 4077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:14.589853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59352 (local address 127.27.185.126:45075)
0929 20:17:14.590081 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:14.590109 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:14.590608 (+ 499us) server_negotiation.cc:407] Connection header received
0929 20:17:14.590809 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:14.590815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:14.590899 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:14.591012 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:14.592374 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.593222 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.594595 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.595002 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.595161 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:14.600357 (+ 5196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:14.600365 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:14.600369 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:14.600415 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:14.604529 (+ 4114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:14.605376 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:14.605382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:14.605386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:14.605528 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:14.605982 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:14.605988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:14.605992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:14.606350 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:14.606636 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:14.607006 (+ 370us) server_negotiation.cc:299] Negotiation successful
0929 20:17:14.607223 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 20:17:14.611721 3531 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:17:14.612522 3531 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:17:14.620229 3674 tablet_service.cc:1467] Processing CreateTablet for tablet 531cc50cb42f4b9fabb0a3b9fb14487e (DEFAULT_TABLE table=test-table [id=5ee87f7c42ae4c6b91e5031dd9d6a8b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:17:14.620456 3947 tablet_service.cc:1467] Processing CreateTablet for tablet 531cc50cb42f4b9fabb0a3b9fb14487e (DEFAULT_TABLE table=test-table [id=5ee87f7c42ae4c6b91e5031dd9d6a8b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:17:14.620697 3674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 531cc50cb42f4b9fabb0a3b9fb14487e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:14.620877 3947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 531cc50cb42f4b9fabb0a3b9fb14487e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:14.624148 3808 tablet_service.cc:1467] Processing CreateTablet for tablet 531cc50cb42f4b9fabb0a3b9fb14487e (DEFAULT_TABLE table=test-table [id=5ee87f7c42ae4c6b91e5031dd9d6a8b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:17:14.624583 3808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 531cc50cb42f4b9fabb0a3b9fb14487e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:14.630843 4041 tablet_bootstrap.cc:492] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd: Bootstrap starting.
I20260929 20:17:14.631204 4042 tablet_bootstrap.cc:492] T 531cc50cb42f4b9fabb0a3b9fb14487e P 57b2ac153bca44e9b3e4e5c82e3eb771: Bootstrap starting.
I20260929 20:17:14.632663 4041 tablet_bootstrap.cc:654] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:14.632885 4042 tablet_bootstrap.cc:654] T 531cc50cb42f4b9fabb0a3b9fb14487e P 57b2ac153bca44e9b3e4e5c82e3eb771: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:14.635576 4043 tablet_bootstrap.cc:492] T 531cc50cb42f4b9fabb0a3b9fb14487e P 5deeb3b1231942048cd724c8f314a9c1: Bootstrap starting.
I20260929 20:17:14.637276 4043 tablet_bootstrap.cc:654] T 531cc50cb42f4b9fabb0a3b9fb14487e P 5deeb3b1231942048cd724c8f314a9c1: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:14.643316 4042 tablet_bootstrap.cc:492] T 531cc50cb42f4b9fabb0a3b9fb14487e P 57b2ac153bca44e9b3e4e5c82e3eb771: No bootstrap required, opened a new log
I20260929 20:17:14.644572 4042 ts_tablet_manager.cc:1397] T 531cc50cb42f4b9fabb0a3b9fb14487e P 57b2ac153bca44e9b3e4e5c82e3eb771: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.001s
I20260929 20:17:14.645407 4042 raft_consensus.cc:348] T 531cc50cb42f4b9fabb0a3b9fb14487e P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:14.645778 4042 raft_consensus.cc:374] T 531cc50cb42f4b9fabb0a3b9fb14487e P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:14.645916 4042 raft_consensus.cc:729] T 531cc50cb42f4b9fabb0a3b9fb14487e P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57b2ac153bca44e9b3e4e5c82e3eb771, State: Initialized, Role: FOLLOWER
I20260929 20:17:14.646201 4042 consensus_queue.cc:260] T 531cc50cb42f4b9fabb0a3b9fb14487e P 57b2ac153bca44e9b3e4e5c82e3eb771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:14.646817 4042 ts_tablet_manager.cc:1428] T 531cc50cb42f4b9fabb0a3b9fb14487e P 57b2ac153bca44e9b3e4e5c82e3eb771: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 20:17:14.650059 4043 tablet_bootstrap.cc:492] T 531cc50cb42f4b9fabb0a3b9fb14487e P 5deeb3b1231942048cd724c8f314a9c1: No bootstrap required, opened a new log
I20260929 20:17:14.650230 4043 ts_tablet_manager.cc:1397] T 531cc50cb42f4b9fabb0a3b9fb14487e P 5deeb3b1231942048cd724c8f314a9c1: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.002s
I20260929 20:17:14.651774 4041 tablet_bootstrap.cc:492] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd: No bootstrap required, opened a new log
I20260929 20:17:14.651965 4041 ts_tablet_manager.cc:1397] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.000s
I20260929 20:17:14.652482 4043 raft_consensus.cc:348] T 531cc50cb42f4b9fabb0a3b9fb14487e P 5deeb3b1231942048cd724c8f314a9c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:14.652601 4041 raft_consensus.cc:348] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:14.652796 4043 raft_consensus.cc:374] T 531cc50cb42f4b9fabb0a3b9fb14487e P 5deeb3b1231942048cd724c8f314a9c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:14.652855 4041 raft_consensus.cc:374] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:14.652910 4043 raft_consensus.cc:729] T 531cc50cb42f4b9fabb0a3b9fb14487e P 5deeb3b1231942048cd724c8f314a9c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5deeb3b1231942048cd724c8f314a9c1, State: Initialized, Role: FOLLOWER
I20260929 20:17:14.652954 4041 raft_consensus.cc:729] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7a9401c551f4b2aa3bb0d32c39a4abd, State: Initialized, Role: FOLLOWER
I20260929 20:17:14.653187 4041 consensus_queue.cc:260] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:14.653200 4043 consensus_queue.cc:260] T 531cc50cb42f4b9fabb0a3b9fb14487e P 5deeb3b1231942048cd724c8f314a9c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:14.653815 4043 ts_tablet_manager.cc:1428] T 531cc50cb42f4b9fabb0a3b9fb14487e P 5deeb3b1231942048cd724c8f314a9c1: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260929 20:17:14.654299 4041 ts_tablet_manager.cc:1428] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:17:14.715613 4048 raft_consensus.cc:482] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:17:14.715871 4048 raft_consensus.cc:504] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:14.717532 4048 leader_election.cc:288] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57b2ac153bca44e9b3e4e5c82e3eb771 (127.27.185.65:43463), 5deeb3b1231942048cd724c8f314a9c1 (127.27.185.67:33609)
I20260929 20:17:14.725939 4089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:14.718172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:41201 (local address 127.27.185.67:33609)
0929 20:17:14.718549 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:14.718555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:14.719117 (+ 562us) server_negotiation.cc:407] Connection header received
0929 20:17:14.719347 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:14.719352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:14.719480 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:14.719595 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:17:14.720265 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.721526 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.723569 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.724807 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.724983 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:14.725080 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:14.725467 (+ 387us) server_negotiation.cc:299] Negotiation successful
0929 20:17:14.725630 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216}
I20260929 20:17:14.726004 4120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:14.717953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:33609 (local address 127.27.185.66:41201)
0929 20:17:14.719035 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:14.719052 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:14.719201 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:14.719610 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:14.719615 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:14.719639 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:17:14.720087 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:14.720097 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.721677 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.721681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:14.723361 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:14.723373 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.725072 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.725078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:14.725257 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:14.725295 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 20:17:14.725421 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 20:17:14.725574 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"mutex_wait_us":39,"thread_start_us":447,"threads_started":1}
I20260929 20:17:14.726956 4037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:14.718156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:55783 (local address 127.27.185.65:43463)
0929 20:17:14.718374 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:14.718382 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:14.718857 (+ 475us) server_negotiation.cc:407] Connection header received
0929 20:17:14.719168 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:14.719174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:14.719261 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:14.719409 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:17:14.720330 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.721452 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.724208 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.725364 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.725575 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:14.725645 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:14.726547 (+ 902us) server_negotiation.cc:299] Negotiation successful
0929 20:17:14.726687 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:17:14.727029 3967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531cc50cb42f4b9fabb0a3b9fb14487e" candidate_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5deeb3b1231942048cd724c8f314a9c1" is_pre_election: true
I20260929 20:17:14.727290 3967 raft_consensus.cc:2454] T 531cc50cb42f4b9fabb0a3b9fb14487e P 5deeb3b1231942048cd724c8f314a9c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7a9401c551f4b2aa3bb0d32c39a4abd in term 0.
I20260929 20:17:14.727821 4121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:14.717795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:43463 (local address 127.27.185.66:55783)
0929 20:17:14.718732 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:14.718754 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:14.718978 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:14.719586 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:14.719592 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:14.719619 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:17:14.720078 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:14.720094 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.721607 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.721613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:14.724041 (+ 2428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:14.724056 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.725804 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.725810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:14.726342 (+ 532us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:14.726379 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 20:17:14.727129 (+ 750us) client_negotiation.cc:240] Negotiation successful
0929 20:17:14.727259 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"mutex_wait_us":2,"thread_start_us":921,"threads_started":1}
I20260929 20:17:14.728113 3762 leader_election.cc:302] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5deeb3b1231942048cd724c8f314a9c1, b7a9401c551f4b2aa3bb0d32c39a4abd; no voters:
I20260929 20:17:14.728541 4048 raft_consensus.cc:2781] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:17:14.728631 4048 raft_consensus.cc:482] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:17:14.728506 3693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531cc50cb42f4b9fabb0a3b9fb14487e" candidate_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" is_pre_election: true
I20260929 20:17:14.728688 4048 raft_consensus.cc:3037] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:17:14.728760 3693 raft_consensus.cc:2454] T 531cc50cb42f4b9fabb0a3b9fb14487e P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7a9401c551f4b2aa3bb0d32c39a4abd in term 0.
I20260929 20:17:14.731182 4048 raft_consensus.cc:504] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:14.731976 4048 leader_election.cc:288] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd [CANDIDATE]: Term 1 election: Requested vote from peers 57b2ac153bca44e9b3e4e5c82e3eb771 (127.27.185.65:43463), 5deeb3b1231942048cd724c8f314a9c1 (127.27.185.67:33609)
I20260929 20:17:14.732968 3693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531cc50cb42f4b9fabb0a3b9fb14487e" candidate_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771"
I20260929 20:17:14.733182 3693 raft_consensus.cc:3037] T 531cc50cb42f4b9fabb0a3b9fb14487e P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:17:14.733248 3967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531cc50cb42f4b9fabb0a3b9fb14487e" candidate_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5deeb3b1231942048cd724c8f314a9c1"
I20260929 20:17:14.733490 3967 raft_consensus.cc:3037] T 531cc50cb42f4b9fabb0a3b9fb14487e P 5deeb3b1231942048cd724c8f314a9c1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:17:14.735710 3693 raft_consensus.cc:2454] T 531cc50cb42f4b9fabb0a3b9fb14487e P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7a9401c551f4b2aa3bb0d32c39a4abd in term 1.
I20260929 20:17:14.736141 3759 leader_election.cc:302] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57b2ac153bca44e9b3e4e5c82e3eb771, b7a9401c551f4b2aa3bb0d32c39a4abd; no voters:
I20260929 20:17:14.736454 4048 raft_consensus.cc:2781] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:17:14.736657 3967 raft_consensus.cc:2454] T 531cc50cb42f4b9fabb0a3b9fb14487e P 5deeb3b1231942048cd724c8f314a9c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7a9401c551f4b2aa3bb0d32c39a4abd in term 1.
I20260929 20:17:14.737303 4048 raft_consensus.cc:686] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd [term 1 LEADER]: Becoming Leader. State: Replica: b7a9401c551f4b2aa3bb0d32c39a4abd, State: Running, Role: LEADER
I20260929 20:17:14.737684 4048 consensus_queue.cc:237] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:14.741242 3532 catalog_manager.cc:5495] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd reported cstate change: term changed from 0 to 1, leader changed from <none> to b7a9401c551f4b2aa3bb0d32c39a4abd (127.27.185.66). New cstate: current_term: 1 leader_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } health_report { overall_health: UNKNOWN } } }
I20260929 20:17:14.762889 4038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:14.756694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42572 (local address 127.27.185.66:43357)
0929 20:17:14.756871 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:14.756876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:14.756893 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:17:14.757130 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:14.757133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:14.757222 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:14.757350 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:17:14.757788 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.758499 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.760515 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:14.760911 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:14.761013 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:14.761608 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:17:14.762261 (+ 653us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:17:14.762409 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:14.762545 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 20:17:14.762617 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:17:14.766746 3693 raft_consensus.cc:1261] T 531cc50cb42f4b9fabb0a3b9fb14487e P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 1 FOLLOWER]: Refusing update from remote peer b7a9401c551f4b2aa3bb0d32c39a4abd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:17:14.766813 3967 raft_consensus.cc:1261] T 531cc50cb42f4b9fabb0a3b9fb14487e P 5deeb3b1231942048cd724c8f314a9c1 [term 1 FOLLOWER]: Refusing update from remote peer b7a9401c551f4b2aa3bb0d32c39a4abd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:17:14.767455 4122 consensus_queue.cc:1035] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd [LEADER]: Connected to new peer: Peer: permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:17:14.767855 4048 consensus_queue.cc:1035] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:17:14.795295 3531 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:17:14.795678 3531 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:17:14.800328 3531 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 424e7e4d77534830b1f0ac96ba2e4982: Sending DeleteTablet for 3 replicas of tablet 531cc50cb42f4b9fabb0a3b9fb14487e
I20260929 20:17:14.801359 3808 tablet_service.cc:1514] Processing DeleteTablet for tablet 531cc50cb42f4b9fabb0a3b9fb14487e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:17:14 UTC) from {username='oryx', principal='oryx/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:42566
I20260929 20:17:14.801693 3674 tablet_service.cc:1514] Processing DeleteTablet for tablet 531cc50cb42f4b9fabb0a3b9fb14487e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:17:14 UTC) from {username='oryx', principal='oryx/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:38230
I20260929 20:17:14.801700 3947 tablet_service.cc:1514] Processing DeleteTablet for tablet 531cc50cb42f4b9fabb0a3b9fb14487e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:17:14 UTC) from {username='oryx', principal='oryx/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:44394
I20260929 20:17:14.801971 4104 tablet_replica.cc:330] T 531cc50cb42f4b9fabb0a3b9fb14487e P 5deeb3b1231942048cd724c8f314a9c1: stopping tablet replica
I20260929 20:17:14.801972 4103 tablet_replica.cc:330] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd: stopping tablet replica
I20260929 20:17:14.802178 4103 raft_consensus.cc:2229] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:17:14.802178 4104 raft_consensus.cc:2229] T 531cc50cb42f4b9fabb0a3b9fb14487e P 5deeb3b1231942048cd724c8f314a9c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:17:14.802453 4104 raft_consensus.cc:2258] T 531cc50cb42f4b9fabb0a3b9fb14487e P 5deeb3b1231942048cd724c8f314a9c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:17:14.802747 4102 tablet_replica.cc:330] T 531cc50cb42f4b9fabb0a3b9fb14487e P 57b2ac153bca44e9b3e4e5c82e3eb771: stopping tablet replica
I20260929 20:17:14.802937 4102 raft_consensus.cc:2229] T 531cc50cb42f4b9fabb0a3b9fb14487e P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:17:14.803164 4102 raft_consensus.cc:2258] T 531cc50cb42f4b9fabb0a3b9fb14487e P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:17:14.804435 4104 ts_tablet_manager.cc:1905] T 531cc50cb42f4b9fabb0a3b9fb14487e P 5deeb3b1231942048cd724c8f314a9c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:17:14.804759 3531 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:17:14.805488 3531 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:17:14.808141 4103 raft_consensus.cc:2258] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:17:14.809819 4103 ts_tablet_manager.cc:1905] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:17:14.814574 3808 tablet_service.cc:1467] Processing CreateTablet for tablet 05d7c6ee0238410aa28462ae04adfd17 (DEFAULT_TABLE table=test-table [id=f058c0780cd54704b2253f9cd59f9d9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:17:14.815062 3808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05d7c6ee0238410aa28462ae04adfd17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:14.815659 3674 tablet_service.cc:1467] Processing CreateTablet for tablet 05d7c6ee0238410aa28462ae04adfd17 (DEFAULT_TABLE table=test-table [id=f058c0780cd54704b2253f9cd59f9d9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:17:14.816082 3674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05d7c6ee0238410aa28462ae04adfd17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:14.820107 3947 tablet_service.cc:1467] Processing CreateTablet for tablet 05d7c6ee0238410aa28462ae04adfd17 (DEFAULT_TABLE table=test-table [id=f058c0780cd54704b2253f9cd59f9d9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:17:14.820575 3947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05d7c6ee0238410aa28462ae04adfd17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:14.824766 4041 tablet_bootstrap.cc:492] T 05d7c6ee0238410aa28462ae04adfd17 P b7a9401c551f4b2aa3bb0d32c39a4abd: Bootstrap starting.
I20260929 20:17:14.826360 4041 tablet_bootstrap.cc:654] T 05d7c6ee0238410aa28462ae04adfd17 P b7a9401c551f4b2aa3bb0d32c39a4abd: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:14.831097 4103 ts_tablet_manager.cc:1918] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:17:14.831275 4103 log.cc:1196] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-1/wal/wals/531cc50cb42f4b9fabb0a3b9fb14487e
I20260929 20:17:14.837765 4042 tablet_bootstrap.cc:492] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771: Bootstrap starting.
I20260929 20:17:14.839541 4042 tablet_bootstrap.cc:654] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:14.843079 4104 ts_tablet_manager.cc:1918] T 531cc50cb42f4b9fabb0a3b9fb14487e P 5deeb3b1231942048cd724c8f314a9c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:17:14.843240 4104 log.cc:1196] T 531cc50cb42f4b9fabb0a3b9fb14487e P 5deeb3b1231942048cd724c8f314a9c1: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-2/wal/wals/531cc50cb42f4b9fabb0a3b9fb14487e
I20260929 20:17:14.844147 4043 tablet_bootstrap.cc:492] T 05d7c6ee0238410aa28462ae04adfd17 P 5deeb3b1231942048cd724c8f314a9c1: Bootstrap starting.
I20260929 20:17:14.843127 4103 ts_tablet_manager.cc:1939] T 531cc50cb42f4b9fabb0a3b9fb14487e P b7a9401c551f4b2aa3bb0d32c39a4abd: Deleting consensus metadata
I20260929 20:17:14.844825 4104 ts_tablet_manager.cc:1939] T 531cc50cb42f4b9fabb0a3b9fb14487e P 5deeb3b1231942048cd724c8f314a9c1: Deleting consensus metadata
I20260929 20:17:14.845923 4043 tablet_bootstrap.cc:654] T 05d7c6ee0238410aa28462ae04adfd17 P 5deeb3b1231942048cd724c8f314a9c1: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:14.845963 3519 catalog_manager.cc:4841] TS 5deeb3b1231942048cd724c8f314a9c1 (127.27.185.67:33609): tablet 531cc50cb42f4b9fabb0a3b9fb14487e (table test-table [id=5ee87f7c42ae4c6b91e5031dd9d6a8b9]) successfully deleted
I20260929 20:17:14.846935 3517 catalog_manager.cc:4841] TS b7a9401c551f4b2aa3bb0d32c39a4abd (127.27.185.66:43357): tablet 531cc50cb42f4b9fabb0a3b9fb14487e (table test-table [id=5ee87f7c42ae4c6b91e5031dd9d6a8b9]) successfully deleted
I20260929 20:17:14.848740 4102 ts_tablet_manager.cc:1905] T 531cc50cb42f4b9fabb0a3b9fb14487e P 57b2ac153bca44e9b3e4e5c82e3eb771: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:17:14.859117 4102 ts_tablet_manager.cc:1918] T 531cc50cb42f4b9fabb0a3b9fb14487e P 57b2ac153bca44e9b3e4e5c82e3eb771: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:17:14.859520 4102 log.cc:1196] T 531cc50cb42f4b9fabb0a3b9fb14487e P 57b2ac153bca44e9b3e4e5c82e3eb771: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-0/wal/wals/531cc50cb42f4b9fabb0a3b9fb14487e
I20260929 20:17:14.860113 4102 ts_tablet_manager.cc:1939] T 531cc50cb42f4b9fabb0a3b9fb14487e P 57b2ac153bca44e9b3e4e5c82e3eb771: Deleting consensus metadata
I20260929 20:17:14.860949 4041 tablet_bootstrap.cc:492] T 05d7c6ee0238410aa28462ae04adfd17 P b7a9401c551f4b2aa3bb0d32c39a4abd: No bootstrap required, opened a new log
I20260929 20:17:14.861132 4041 ts_tablet_manager.cc:1397] T 05d7c6ee0238410aa28462ae04adfd17 P b7a9401c551f4b2aa3bb0d32c39a4abd: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.000s
I20260929 20:17:14.861970 4043 tablet_bootstrap.cc:492] T 05d7c6ee0238410aa28462ae04adfd17 P 5deeb3b1231942048cd724c8f314a9c1: No bootstrap required, opened a new log
I20260929 20:17:14.862167 4043 ts_tablet_manager.cc:1397] T 05d7c6ee0238410aa28462ae04adfd17 P 5deeb3b1231942048cd724c8f314a9c1: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.009s
I20260929 20:17:14.862068 4041 raft_consensus.cc:348] T 05d7c6ee0238410aa28462ae04adfd17 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:14.862383 4041 raft_consensus.cc:374] T 05d7c6ee0238410aa28462ae04adfd17 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:14.862496 4041 raft_consensus.cc:729] T 05d7c6ee0238410aa28462ae04adfd17 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7a9401c551f4b2aa3bb0d32c39a4abd, State: Initialized, Role: FOLLOWER
I20260929 20:17:14.862991 4041 consensus_queue.cc:260] T 05d7c6ee0238410aa28462ae04adfd17 P b7a9401c551f4b2aa3bb0d32c39a4abd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:14.863062 4043 raft_consensus.cc:348] T 05d7c6ee0238410aa28462ae04adfd17 P 5deeb3b1231942048cd724c8f314a9c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:14.863318 4043 raft_consensus.cc:374] T 05d7c6ee0238410aa28462ae04adfd17 P 5deeb3b1231942048cd724c8f314a9c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:14.863421 4043 raft_consensus.cc:729] T 05d7c6ee0238410aa28462ae04adfd17 P 5deeb3b1231942048cd724c8f314a9c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5deeb3b1231942048cd724c8f314a9c1, State: Initialized, Role: FOLLOWER
I20260929 20:17:14.863528 4041 ts_tablet_manager.cc:1428] T 05d7c6ee0238410aa28462ae04adfd17 P b7a9401c551f4b2aa3bb0d32c39a4abd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:17:14.863674 4043 consensus_queue.cc:260] T 05d7c6ee0238410aa28462ae04adfd17 P 5deeb3b1231942048cd724c8f314a9c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:14.864219 4043 ts_tablet_manager.cc:1428] T 05d7c6ee0238410aa28462ae04adfd17 P 5deeb3b1231942048cd724c8f314a9c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:17:14.865914 3516 catalog_manager.cc:4841] TS 57b2ac153bca44e9b3e4e5c82e3eb771 (127.27.185.65:43463): tablet 531cc50cb42f4b9fabb0a3b9fb14487e (table test-table [id=5ee87f7c42ae4c6b91e5031dd9d6a8b9]) successfully deleted
I20260929 20:17:14.866788 4042 tablet_bootstrap.cc:492] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771: No bootstrap required, opened a new log
I20260929 20:17:14.866957 4042 ts_tablet_manager.cc:1397] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.016s
I20260929 20:17:14.867767 4042 raft_consensus.cc:348] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:14.868042 4042 raft_consensus.cc:374] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:14.868144 4042 raft_consensus.cc:729] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57b2ac153bca44e9b3e4e5c82e3eb771, State: Initialized, Role: FOLLOWER
I20260929 20:17:14.868395 4042 consensus_queue.cc:260] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:14.868914 4042 ts_tablet_manager.cc:1428] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 20:17:15.161213 4098 raft_consensus.cc:482] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:17:15.161398 4098 raft_consensus.cc:504] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:15.162197 4098 leader_election.cc:288] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7a9401c551f4b2aa3bb0d32c39a4abd (127.27.185.66:43357), 5deeb3b1231942048cd724c8f314a9c1 (127.27.185.67:33609)
I20260929 20:17:15.162789 3828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d7c6ee0238410aa28462ae04adfd17" candidate_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" is_pre_election: true
I20260929 20:17:15.162885 3967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d7c6ee0238410aa28462ae04adfd17" candidate_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5deeb3b1231942048cd724c8f314a9c1" is_pre_election: true
I20260929 20:17:15.163097 3828 raft_consensus.cc:2454] T 05d7c6ee0238410aa28462ae04adfd17 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57b2ac153bca44e9b3e4e5c82e3eb771 in term 0.
I20260929 20:17:15.163120 3967 raft_consensus.cc:2454] T 05d7c6ee0238410aa28462ae04adfd17 P 5deeb3b1231942048cd724c8f314a9c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57b2ac153bca44e9b3e4e5c82e3eb771 in term 0.
I20260929 20:17:15.163668 3622 leader_election.cc:302] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 57b2ac153bca44e9b3e4e5c82e3eb771, 5deeb3b1231942048cd724c8f314a9c1, b7a9401c551f4b2aa3bb0d32c39a4abd; no voters:
I20260929 20:17:15.164062 4098 raft_consensus.cc:2781] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:17:15.164197 4098 raft_consensus.cc:482] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:17:15.164304 4098 raft_consensus.cc:3037] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:17:15.167464 4098 raft_consensus.cc:504] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:15.168145 4098 leader_election.cc:288] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771 [CANDIDATE]: Term 1 election: Requested vote from peers b7a9401c551f4b2aa3bb0d32c39a4abd (127.27.185.66:43357), 5deeb3b1231942048cd724c8f314a9c1 (127.27.185.67:33609)
I20260929 20:17:15.168675 3828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d7c6ee0238410aa28462ae04adfd17" candidate_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd"
I20260929 20:17:15.168680 3967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d7c6ee0238410aa28462ae04adfd17" candidate_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5deeb3b1231942048cd724c8f314a9c1"
I20260929 20:17:15.168888 3828 raft_consensus.cc:3037] T 05d7c6ee0238410aa28462ae04adfd17 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:17:15.168893 3967 raft_consensus.cc:3037] T 05d7c6ee0238410aa28462ae04adfd17 P 5deeb3b1231942048cd724c8f314a9c1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:17:15.171176 3828 raft_consensus.cc:2454] T 05d7c6ee0238410aa28462ae04adfd17 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57b2ac153bca44e9b3e4e5c82e3eb771 in term 1.
I20260929 20:17:15.171730 3620 leader_election.cc:302] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57b2ac153bca44e9b3e4e5c82e3eb771, b7a9401c551f4b2aa3bb0d32c39a4abd; no voters:
I20260929 20:17:15.171880 3967 raft_consensus.cc:2454] T 05d7c6ee0238410aa28462ae04adfd17 P 5deeb3b1231942048cd724c8f314a9c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57b2ac153bca44e9b3e4e5c82e3eb771 in term 1.
I20260929 20:17:15.172048 4098 raft_consensus.cc:2781] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:17:15.174546 4098 raft_consensus.cc:686] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 1 LEADER]: Becoming Leader. State: Replica: 57b2ac153bca44e9b3e4e5c82e3eb771, State: Running, Role: LEADER
I20260929 20:17:15.174867 4098 consensus_queue.cc:237] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } }
I20260929 20:17:15.177802 3532 catalog_manager.cc:5495] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771 reported cstate change: term changed from 0 to 1, leader changed from <none> to 57b2ac153bca44e9b3e4e5c82e3eb771 (127.27.185.65). New cstate: current_term: 1 leader_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57b2ac153bca44e9b3e4e5c82e3eb771" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 43463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 } health_report { overall_health: UNKNOWN } } }
I20260929 20:17:15.208364 4037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:15.202655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38248 (local address 127.27.185.65:43463)
0929 20:17:15.203170 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:15.203175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:15.203188 (+ 13us) server_negotiation.cc:407] Connection header received
0929 20:17:15.203235 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:15.203238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:15.203294 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:15.203371 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:17:15.203866 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:15.204571 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:15.206647 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:15.207041 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:15.207133 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:15.207677 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:17:15.207781 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:17:15.207848 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:15.208030 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 20:17:15.208090 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419}
I20260929 20:17:15.218037 3967 raft_consensus.cc:1261] T 05d7c6ee0238410aa28462ae04adfd17 P 5deeb3b1231942048cd724c8f314a9c1 [term 1 FOLLOWER]: Refusing update from remote peer 57b2ac153bca44e9b3e4e5c82e3eb771: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:17:15.218178 3828 raft_consensus.cc:1261] T 05d7c6ee0238410aa28462ae04adfd17 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 1 FOLLOWER]: Refusing update from remote peer 57b2ac153bca44e9b3e4e5c82e3eb771: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:17:15.218719 4132 consensus_queue.cc:1035] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5deeb3b1231942048cd724c8f314a9c1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 33609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:17:15.219183 4132 consensus_queue.cc:1035] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a9401c551f4b2aa3bb0d32c39a4abd" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:17:15.226790 4056 mvcc.cc:205] Tried to move back new op lower bound from 7334760592247398400 to 7334760592078999552. Current Snapshot: MvccSnapshot[applied={T|T < 7334760592247398400}]
I20260929 20:17:15.269475 3532 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:17:15.269688 3532 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:17:15.272543 3532 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 424e7e4d77534830b1f0ac96ba2e4982: Sending DeleteTablet for 3 replicas of tablet 05d7c6ee0238410aa28462ae04adfd17
I20260929 20:17:15.273311 3674 tablet_service.cc:1514] Processing DeleteTablet for tablet 05d7c6ee0238410aa28462ae04adfd17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:17:15 UTC) from {username='oryx', principal='oryx/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:38230
I20260929 20:17:15.273391 3808 tablet_service.cc:1514] Processing DeleteTablet for tablet 05d7c6ee0238410aa28462ae04adfd17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:17:15 UTC) from {username='oryx', principal='oryx/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:42566
I20260929 20:17:15.273633 4102 tablet_replica.cc:330] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771: stopping tablet replica
I20260929 20:17:15.273619 3947 tablet_service.cc:1514] Processing DeleteTablet for tablet 05d7c6ee0238410aa28462ae04adfd17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:17:15 UTC) from {username='oryx', principal='oryx/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:44394
I20260929 20:17:15.273722 4103 tablet_replica.cc:330] T 05d7c6ee0238410aa28462ae04adfd17 P b7a9401c551f4b2aa3bb0d32c39a4abd: stopping tablet replica
I20260929 20:17:15.273818 4102 raft_consensus.cc:2229] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:17:15.273904 4104 tablet_replica.cc:330] T 05d7c6ee0238410aa28462ae04adfd17 P 5deeb3b1231942048cd724c8f314a9c1: stopping tablet replica
I20260929 20:17:15.273890 4103 raft_consensus.cc:2229] T 05d7c6ee0238410aa28462ae04adfd17 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:17:15.274077 4104 raft_consensus.cc:2229] T 05d7c6ee0238410aa28462ae04adfd17 P 5deeb3b1231942048cd724c8f314a9c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:17:15.274096 4103 raft_consensus.cc:2258] T 05d7c6ee0238410aa28462ae04adfd17 P b7a9401c551f4b2aa3bb0d32c39a4abd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:17:15.274171 4102 raft_consensus.cc:2258] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:17:15.274286 4104 raft_consensus.cc:2258] T 05d7c6ee0238410aa28462ae04adfd17 P 5deeb3b1231942048cd724c8f314a9c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:17:15.275254 4103 ts_tablet_manager.cc:1905] T 05d7c6ee0238410aa28462ae04adfd17 P b7a9401c551f4b2aa3bb0d32c39a4abd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:17:15.275341 4102 ts_tablet_manager.cc:1905] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:17:15.275355 4104 ts_tablet_manager.cc:1905] T 05d7c6ee0238410aa28462ae04adfd17 P 5deeb3b1231942048cd724c8f314a9c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:17:15.281376 4104 ts_tablet_manager.cc:1918] T 05d7c6ee0238410aa28462ae04adfd17 P 5deeb3b1231942048cd724c8f314a9c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:17:15.281534 4104 log.cc:1196] T 05d7c6ee0238410aa28462ae04adfd17 P 5deeb3b1231942048cd724c8f314a9c1: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-2/wal/wals/05d7c6ee0238410aa28462ae04adfd17
I20260929 20:17:15.281910 4104 ts_tablet_manager.cc:1939] T 05d7c6ee0238410aa28462ae04adfd17 P 5deeb3b1231942048cd724c8f314a9c1: Deleting consensus metadata
I20260929 20:17:15.281986 4102 ts_tablet_manager.cc:1918] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:17:15.282130 4102 log.cc:1196] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-0/wal/wals/05d7c6ee0238410aa28462ae04adfd17
I20260929 20:17:15.282541 4102 ts_tablet_manager.cc:1939] T 05d7c6ee0238410aa28462ae04adfd17 P 57b2ac153bca44e9b3e4e5c82e3eb771: Deleting consensus metadata
I20260929 20:17:15.283061 4103 ts_tablet_manager.cc:1918] T 05d7c6ee0238410aa28462ae04adfd17 P b7a9401c551f4b2aa3bb0d32c39a4abd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:17:15.283026 3519 catalog_manager.cc:4841] TS 5deeb3b1231942048cd724c8f314a9c1 (127.27.185.67:33609): tablet 05d7c6ee0238410aa28462ae04adfd17 (table test-table [id=f058c0780cd54704b2253f9cd59f9d9f]) successfully deleted
I20260929 20:17:15.283200 4103 log.cc:1196] T 05d7c6ee0238410aa28462ae04adfd17 P b7a9401c551f4b2aa3bb0d32c39a4abd: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790712966917858-28389-0/minicluster-data/ts-1/wal/wals/05d7c6ee0238410aa28462ae04adfd17
I20260929 20:17:15.283634 4103 ts_tablet_manager.cc:1939] T 05d7c6ee0238410aa28462ae04adfd17 P b7a9401c551f4b2aa3bb0d32c39a4abd: Deleting consensus metadata
I20260929 20:17:15.283710 3516 catalog_manager.cc:4841] TS 57b2ac153bca44e9b3e4e5c82e3eb771 (127.27.185.65:43463): tablet 05d7c6ee0238410aa28462ae04adfd17 (table test-table [id=f058c0780cd54704b2253f9cd59f9d9f]) successfully deleted
I20260929 20:17:15.284164 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 3603
I20260929 20:17:15.284566 3517 catalog_manager.cc:4841] TS b7a9401c551f4b2aa3bb0d32c39a4abd (127.27.185.66:43357): tablet 05d7c6ee0238410aa28462ae04adfd17 (table test-table [id=f058c0780cd54704b2253f9cd59f9d9f]) successfully deleted
Sep 29 20:17:15 dist-test-slave-4cx3 krb5kdc[3487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.185.65@KRBTEST.COM
I20260929 20:17:15.298864 4037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:15.285111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38250 (local address 127.27.185.65:43463)
0929 20:17:15.285286 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:15.285291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:15.285645 (+ 354us) server_negotiation.cc:407] Connection header received
0929 20:17:15.285789 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:15.285792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:15.285870 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:15.285949 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:15.287147 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:15.287971 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:15.289185 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:15.289595 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:15.289685 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:15.293475 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:15.293490 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:15.293506 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:15.293539 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:15.296522 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:15.297146 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:15.297153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:15.297156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:15.297299 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:15.297647 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:15.297650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:15.297652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:15.298007 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:15.298231 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:15.298465 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 20:17:15.298629 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:17:15.299638 3734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:15.426100 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 3603
I20260929 20:17:15.448607 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 3743
Sep 29 20:17:15 dist-test-slave-4cx3 krb5kdc[3487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.185.66@KRBTEST.COM
I20260929 20:17:15.463346 4143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:15.449381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42586 (local address 127.27.185.66:43357)
0929 20:17:15.449831 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:15.449836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:15.449851 (+ 15us) server_negotiation.cc:407] Connection header received
0929 20:17:15.449906 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:15.449910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:15.449976 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:15.450073 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:15.451248 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:15.451968 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:15.453185 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:15.453576 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:15.453664 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:15.457768 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:15.457783 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:15.457798 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:15.457831 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:15.460569 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:15.461277 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:15.461284 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:15.461286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:15.461408 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:15.461776 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:15.461779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:15.461781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:15.462176 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:15.462441 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:15.462780 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 20:17:15.462995 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":240,"threads_started":1}
I20260929 20:17:15.464120 3873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:15.587596 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 3743
I20260929 20:17:15.610105 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 3882
Sep 29 20:17:15 dist-test-slave-4cx3 krb5kdc[3487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.185.67@KRBTEST.COM
I20260929 20:17:15.625756 4145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:15.610894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44416 (local address 127.27.185.67:33609)
0929 20:17:15.611670 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:15.611675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:15.611691 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:17:15.611740 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:15.611743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:15.611813 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:15.611938 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:15.613523 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:15.614258 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:15.615587 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:15.615996 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:15.616142 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:15.620443 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:15.620459 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:15.620475 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:15.620522 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:15.623157 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:15.623914 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:15.623922 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:15.623924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:15.624063 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:15.624383 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:15.624387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:15.624389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:15.624740 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:15.624944 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:15.625215 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 20:17:15.625380 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":556,"threads_started":1}
I20260929 20:17:15.626461 4012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:15.749159 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 3882
I20260929 20:17:15.770325 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 3500
I20260929 20:17:15.771581 3572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:15.888592 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 3500
2026-09-29T20:17:15Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5831 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:17:15 dist-test-slave-4cx3 krb5kdc[4151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:17:15 dist-test-slave-4cx3 krb5kdc[4151](info): set up 2 sockets
Sep 29 20:17:15 dist-test-slave-4cx3 krb5kdc[4151](info): commencing operation
krb5kdc: starting...
W20260929 20:17:16.995853 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:17:17 dist-test-slave-4cx3 krb5kdc[4151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:17:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:17:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
I20260929 20:17:17.227757 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:42327
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:38409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:42327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:17:17.557530 4164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:17.557830 4164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:17.557919 4164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:17.565318 4164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:17:17.565461 4164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:17.565510 4164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:17.565560 4164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:17:17.565613 4164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:17:17.575688 4164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:38409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:42327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:42327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:17.576959 4164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:17.578200 4164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:17.590893 4169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:17.591048 4172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:17.591089 4170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:17.592324 4164 server_base.cc:1034] running on GCE node
I20260929 20:17:17.593050 4164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:17.594341 4164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:17.595497 4164 hybrid_clock.cc:648] HybridClock initialized: now 1790713037595458 us; error 27 us; skew 500 ppm
I20260929 20:17:17.595853 4164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T20:17:17Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1777 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:17:17 dist-test-slave-4cx3 krb5kdc[4178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:17:17 dist-test-slave-4cx3 krb5kdc[4178](info): set up 2 sockets
Sep 29 20:17:17 dist-test-slave-4cx3 krb5kdc[4178](info): commencing operation
krb5kdc: starting...
W20260929 20:17:18.779457 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.001s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:17:18 dist-test-slave-4cx3 krb5kdc[4178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:17:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:17:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
I20260929 20:17:18.999336 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:39341
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:39341
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:17:19.326418 4191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:19.326704 4191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:19.326860 4191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:19.334117 4191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:17:19.334219 4191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:19.334304 4191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:19.334390 4191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:17:19.334466 4191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:17:19.344198 4191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:39341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:39341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:19.345525 4191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:19.346859 4191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:19.358836 4196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:19.358884 4199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:19.358929 4197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:19.360080 4191 server_base.cc:1034] running on GCE node
I20260929 20:17:19.360869 4191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:19.362275 4191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:19.363502 4191 hybrid_clock.cc:648] HybridClock initialized: now 1790713039363456 us; error 37 us; skew 500 ppm
I20260929 20:17:19.363847 4191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:17:19 dist-test-slave-4cx3 krb5kdc[4178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713039, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:17:19.367058 4191 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:17:19.368844 4191 webserver.cc:460] Webserver started at http://127.27.185.126:37887/ using document root <none> and password file <none>
I20260929 20:17:19.369652 4191 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:19.369832 4191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:19.370175 4191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:17:19.373533 4191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "18ab230587d244f8a12fd275daadfbbe"
format_stamp: "Formatted at 2026-09-29 20:17:19 on dist-test-slave-4cx3"
server_key: "45692c7a8225dff1fc402e66bd9a274d"
server_key_iv: "e41d5716a342e426c496ac258609a9a8"
server_key_version: "encryptionkey@0"
I20260929 20:17:19.374368 4191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "18ab230587d244f8a12fd275daadfbbe"
format_stamp: "Formatted at 2026-09-29 20:17:19 on dist-test-slave-4cx3"
server_key: "45692c7a8225dff1fc402e66bd9a274d"
server_key_iv: "e41d5716a342e426c496ac258609a9a8"
server_key_version: "encryptionkey@0"
I20260929 20:17:19.380167 4191 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:17:19.383857 4206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:19.385437 4191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 20:17:19.385667 4191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "18ab230587d244f8a12fd275daadfbbe"
format_stamp: "Formatted at 2026-09-29 20:17:19 on dist-test-slave-4cx3"
server_key: "45692c7a8225dff1fc402e66bd9a274d"
server_key_iv: "e41d5716a342e426c496ac258609a9a8"
server_key_version: "encryptionkey@0"
I20260929 20:17:19.385908 4191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:19.413856 4191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:19.418277 4191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:19.418607 4191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:19.439124 4191 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:39341
I20260929 20:17:19.439143 4257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:39341 every 8 connection(s)
I20260929 20:17:19.440678 4191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:17:19.444475 4258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:19.451045 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 4191
I20260929 20:17:19.451287 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:17:19.452100 28389 external_mini_cluster.cc:1428] Setting key 6f430650a80ff5dbd66a044c97b00d67
I20260929 20:17:19.455874 4258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe: Bootstrap starting.
I20260929 20:17:19.459400 4258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:19.460587 4258 log.cc:824] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe: Log is configured to *not* fsync() on all Append() calls
I20260929 20:17:19.463522 4258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe: No bootstrap required, opened a new log
Sep 29 20:17:19 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:17:19.472070 4258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ab230587d244f8a12fd275daadfbbe" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 39341 } }
I20260929 20:17:19.472500 4258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:19.472620 4258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18ab230587d244f8a12fd275daadfbbe, State: Initialized, Role: FOLLOWER
I20260929 20:17:19.473325 4258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ab230587d244f8a12fd275daadfbbe" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 39341 } }
I20260929 20:17:19.473660 4258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:17:19.473809 4258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:17:19.474013 4258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:17:19.476157 4261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:19.454252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41012 (local address 127.27.185.126:39341)
0929 20:17:19.455060 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:19.455073 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:19.455114 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:17:19.456969 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:19.456991 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:19.457372 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:19.457721 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:19.459580 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:19.461008 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:19.462584 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:19.463223 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:19.463345 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:19.468005 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:19.468014 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:19.468035 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:19.468076 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:19.471327 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:19.472072 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:19.472079 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:19.472085 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:19.472253 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:19.472730 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:19.472736 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:19.472739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:19.473465 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:19.473787 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:19.474272 (+ 485us) server_negotiation.cc:299] Negotiation successful
0929 20:17:19.474736 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":386,"threads_started":1}
I20260929 20:17:19.477341 4258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ab230587d244f8a12fd275daadfbbe" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 39341 } }
I20260929 20:17:19.477917 4258 leader_election.cc:302] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18ab230587d244f8a12fd275daadfbbe; no voters:
I20260929 20:17:19.478510 4258 leader_election.cc:288] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:17:19.478664 4263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:17:19.479770 4263 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [term 1 LEADER]: Becoming Leader. State: Replica: 18ab230587d244f8a12fd275daadfbbe, State: Running, Role: LEADER
I20260929 20:17:19.480255 4263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ab230587d244f8a12fd275daadfbbe" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 39341 } }
I20260929 20:17:19.480657 4258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:17:19.483076 4265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18ab230587d244f8a12fd275daadfbbe. Latest consensus state: current_term: 1 leader_uuid: "18ab230587d244f8a12fd275daadfbbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ab230587d244f8a12fd275daadfbbe" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 39341 } } }
I20260929 20:17:19.483198 4264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18ab230587d244f8a12fd275daadfbbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ab230587d244f8a12fd275daadfbbe" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 39341 } } }
I20260929 20:17:19.483531 4264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [sys.catalog]: This master's current role is: LEADER
I20260929 20:17:19.483531 4265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [sys.catalog]: This master's current role is: LEADER
I20260929 20:17:19.484342 4269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:17:19.490751 4269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:17:19.502395 4269 catalog_manager.cc:1347] Generated new cluster ID: 2effacc342c74dffbc2e110be8b0c12a
I20260929 20:17:19.502547 4269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:17:19.532677 4269 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:17:19.533959 4269 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:17:19.541322 4269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe: Generated new TSK 0
I20260929 20:17:19.542449 4269 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
I20260929 20:17:19.603317 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:0
--local_ip_for_outbound_sockets=127.27.185.65
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:39341
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:17:19.937132 4284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:19.937455 4284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:19.937628 4284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:19.944730 4284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:19.944886 4284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:19.945132 4284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:17:19.954828 4284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:39341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:19.956225 4284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:19.957512 4284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:19.970996 4289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:19.971053 4290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:19.971109 4292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:19.972281 4284 server_base.cc:1034] running on GCE node
I20260929 20:17:19.972934 4284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:19.973840 4284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:19.975042 4284 hybrid_clock.cc:648] HybridClock initialized: now 1790713039974995 us; error 34 us; skew 500 ppm
I20260929 20:17:19.975371 4284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:17:19 dist-test-slave-4cx3 krb5kdc[4178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713039, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:17:19.978536 4284 init.cc:377] Logged in from keytab as kudu/127.27.185.65@KRBTEST.COM (short username kudu)
I20260929 20:17:19.980108 4284 webserver.cc:460] Webserver started at http://127.27.185.65:42245/ using document root <none> and password file <none>
I20260929 20:17:19.980831 4284 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:19.980957 4284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:19.981366 4284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:17:19.984673 4284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-0/data/instance:
uuid: "d0272ab77d9c45139a572b7672f61f75"
format_stamp: "Formatted at 2026-09-29 20:17:19 on dist-test-slave-4cx3"
server_key: "3f8068ff83988974947db94b021d5e6d"
server_key_iv: "7f1e1f56f4ba7c27cc45cc093443b5cf"
server_key_version: "encryptionkey@0"
I20260929 20:17:19.985504 4284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance:
uuid: "d0272ab77d9c45139a572b7672f61f75"
format_stamp: "Formatted at 2026-09-29 20:17:19 on dist-test-slave-4cx3"
server_key: "3f8068ff83988974947db94b021d5e6d"
server_key_iv: "7f1e1f56f4ba7c27cc45cc093443b5cf"
server_key_version: "encryptionkey@0"
I20260929 20:17:19.990976 4284 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 20:17:19.994696 4299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:19.996107 4284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:17:19.996284 4284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "d0272ab77d9c45139a572b7672f61f75"
format_stamp: "Formatted at 2026-09-29 20:17:19 on dist-test-slave-4cx3"
server_key: "3f8068ff83988974947db94b021d5e6d"
server_key_iv: "7f1e1f56f4ba7c27cc45cc093443b5cf"
server_key_version: "encryptionkey@0"
I20260929 20:17:19.996480 4284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:20.011013 4284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:20.015105 4284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:20.015491 4284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:20.016561 4284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:17:20.017884 4284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:17:20.017957 4284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:20.018133 4284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:17:20.018198 4284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:20.051894 4284 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:36773
I20260929 20:17:20.051936 4411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:36773 every 8 connection(s)
I20260929 20:17:20.053190 4284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:17:20.057349 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 4284
I20260929 20:17:20.057576 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance
I20260929 20:17:20.058439 28389 external_mini_cluster.cc:1428] Setting key 15aa42d5a9b2a35ebe57936128377447
Sep 29 20:17:20 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713039, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:17:20.077054 4414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:20.055840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:42937 (local address 127.27.185.126:39341)
0929 20:17:20.056272 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:20.056276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:20.057414 (+ 1138us) server_negotiation.cc:407] Connection header received
0929 20:17:20.059870 (+ 2456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:20.059875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:20.059938 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:20.060022 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:20.061956 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:20.062674 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:20.065127 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:20.065543 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:20.065647 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:20.070711 (+ 5064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:20.070717 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:20.070720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:20.070755 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:20.073526 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:20.074581 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:20.074586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:20.074588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:20.074721 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:20.075227 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:20.075230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:20.075232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:20.075581 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:20.075696 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:20.076612 (+ 916us) server_negotiation.cc:299] Negotiation successful
0929 20:17:20.076793 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":207,"threads_started":1}
I20260929 20:17:20.078743 4415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:20.056418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.65:42937)
0929 20:17:20.057212 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:20.057266 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:20.059561 (+ 2295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:20.060303 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:20.060312 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:20.060977 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:20.061775 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:20.061803 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:20.062851 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:20.062859 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:20.064971 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:20.064988 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:20.065776 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:20.065785 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:20.066117 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:20.066866 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:20.066901 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:20.070417 (+ 3516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:20.073693 (+ 3276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:20.073705 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:20.073726 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:20.074439 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:20.074835 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:20.074843 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:20.074849 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:20.075112 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:20.075716 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:20.075728 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:20.076217 (+ 489us) client_negotiation.cc:769] Sending connection context
0929 20:17:20.076534 (+ 317us) client_negotiation.cc:240] Negotiation successful
0929 20:17:20.077167 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":265,"threads_started":1}
I20260929 20:17:20.080623 4412 heartbeater.cc:347] Connected to a master server at 127.27.185.126:39341
I20260929 20:17:20.080986 4412 heartbeater.cc:464] Registering TS with master...
I20260929 20:17:20.081957 4412 heartbeater.cc:511] Master 127.27.185.126:39341 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.185.66@KRBTEST.COM; defaulting to no policy
I20260929 20:17:20.084460 4223 ts_manager.cc:194] Registered new tserver with Master: d0272ab77d9c45139a572b7672f61f75 (127.27.185.65:36773)
I20260929 20:17:20.086161 4223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.65@KRBTEST.COM'} at 127.27.185.65:42937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
I20260929 20:17:20.114676 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:0
--local_ip_for_outbound_sockets=127.27.185.66
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:39341
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:17:20.436896 4418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:20.437173 4418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:20.437340 4418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:20.445088 4418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:20.445235 4418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:20.445425 4418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:17:20.455091 4418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:39341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:20.456409 4418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:20.457716 4418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:20.470785 4423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:20.470842 4426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:20.470865 4424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:20.471570 4418 server_base.cc:1034] running on GCE node
I20260929 20:17:20.472178 4418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:20.473145 4418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:20.474431 4418 hybrid_clock.cc:648] HybridClock initialized: now 1790713040474354 us; error 61 us; skew 500 ppm
I20260929 20:17:20.474790 4418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:17:20 dist-test-slave-4cx3 krb5kdc[4178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713040, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:17:20.478111 4418 init.cc:377] Logged in from keytab as kudu/127.27.185.66@KRBTEST.COM (short username kudu)
I20260929 20:17:20.479729 4418 webserver.cc:460] Webserver started at http://127.27.185.66:33015/ using document root <none> and password file <none>
I20260929 20:17:20.480449 4418 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:20.480597 4418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:20.480882 4418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:17:20.483870 4418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-1/data/instance:
uuid: "3af90c3ef7c54392b87acb8049373bb7"
format_stamp: "Formatted at 2026-09-29 20:17:20 on dist-test-slave-4cx3"
server_key: "c429aea971dfaa0e4e383243c69be6c1"
server_key_iv: "6ee09048f716c736270d11b8da02309a"
server_key_version: "encryptionkey@0"
I20260929 20:17:20.484675 4418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance:
uuid: "3af90c3ef7c54392b87acb8049373bb7"
format_stamp: "Formatted at 2026-09-29 20:17:20 on dist-test-slave-4cx3"
server_key: "c429aea971dfaa0e4e383243c69be6c1"
server_key_iv: "6ee09048f716c736270d11b8da02309a"
server_key_version: "encryptionkey@0"
I20260929 20:17:20.490233 4418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:17:20.493831 4433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:20.495245 4418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 20:17:20.495440 4418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "3af90c3ef7c54392b87acb8049373bb7"
format_stamp: "Formatted at 2026-09-29 20:17:20 on dist-test-slave-4cx3"
server_key: "c429aea971dfaa0e4e383243c69be6c1"
server_key_iv: "6ee09048f716c736270d11b8da02309a"
server_key_version: "encryptionkey@0"
I20260929 20:17:20.495652 4418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:20.523561 4418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:20.527714 4418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:20.528019 4418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:20.528973 4418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:17:20.530328 4418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:17:20.530401 4418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:20.530565 4418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:17:20.530616 4418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:20.564023 4418 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:36251
I20260929 20:17:20.564111 4545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:36251 every 8 connection(s)
I20260929 20:17:20.565371 4418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:17:20.565876 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 4418
I20260929 20:17:20.566087 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance
I20260929 20:17:20.566696 28389 external_mini_cluster.cc:1428] Setting key ee0384835bf5802464121869ecb1cceb
Sep 29 20:17:20 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713040, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:17:20.587378 4414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:20.568342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:53075 (local address 127.27.185.126:39341)
0929 20:17:20.568555 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:20.568561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:20.570115 (+ 1554us) server_negotiation.cc:407] Connection header received
0929 20:17:20.572263 (+ 2148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:20.572269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:20.572374 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:20.572494 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:20.574235 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:20.574983 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:20.576739 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:20.577194 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:20.577335 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:20.581175 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:20.581180 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:20.581183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:20.581220 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:20.583938 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:20.584733 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:20.584739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:20.584741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:20.584876 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:20.585396 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:20.585402 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:20.585407 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:20.585760 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:20.585937 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:20.586753 (+ 816us) server_negotiation.cc:299] Negotiation successful
0929 20:17:20.587029 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.27.185.67@KRBTEST.COM; defaulting to no policy
I20260929 20:17:20.588526 4548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:20.569070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.66:53075)
0929 20:17:20.569926 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:20.569977 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:20.571934 (+ 1957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:20.572765 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:20.572775 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:20.573384 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:20.574029 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:20.574048 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:20.575171 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:20.575176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:20.576571 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:20.576589 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:20.577452 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:20.577459 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:20.577700 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:20.578428 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:20.578454 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:20.580918 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:20.584091 (+ 3173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:20.584101 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:20.584120 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:20.584585 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:20.585002 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:20.585007 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:20.585011 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:20.585262 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:20.585894 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:20.585903 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:20.586282 (+ 379us) client_negotiation.cc:769] Sending connection context
0929 20:17:20.586652 (+ 370us) client_negotiation.cc:240] Negotiation successful
0929 20:17:20.587241 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":391,"threads_started":1}
I20260929 20:17:20.590294 4546 heartbeater.cc:347] Connected to a master server at 127.27.185.126:39341
I20260929 20:17:20.590720 4546 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.67@KRBTEST.COM" created.
I20260929 20:17:20.591689 4546 heartbeater.cc:511] Master 127.27.185.126:39341 requested a full tablet report, sending...
I20260929 20:17:20.593214 4223 ts_manager.cc:194] Registered new tserver with Master: 3af90c3ef7c54392b87acb8049373bb7 (127.27.185.66:36251)
I20260929 20:17:20.594048 4223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.66@KRBTEST.COM'} at 127.27.185.66:53075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
I20260929 20:17:20.618767 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:0
--local_ip_for_outbound_sockets=127.27.185.67
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:39341
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:17:20.944810 4551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:20.945101 4551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:20.945195 4551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:20.952306 4551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:20.952420 4551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:20.952529 4551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:17:20.962563 4551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:39341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:20.963936 4551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:20.965255 4551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:20.979008 4556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:20.978945 4557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:20.978957 4559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:20.980080 4551 server_base.cc:1034] running on GCE node
I20260929 20:17:20.980705 4551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:20.981717 4551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:20.982957 4551 hybrid_clock.cc:648] HybridClock initialized: now 1790713040982876 us; error 74 us; skew 500 ppm
I20260929 20:17:20.983299 4551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:17:20 dist-test-slave-4cx3 krb5kdc[4178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713040, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:17:20.986744 4551 init.cc:377] Logged in from keytab as kudu/127.27.185.67@KRBTEST.COM (short username kudu)
I20260929 20:17:20.988278 4551 webserver.cc:460] Webserver started at http://127.27.185.67:46185/ using document root <none> and password file <none>
I20260929 20:17:20.989001 4551 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:20.989143 4551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:20.989471 4551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:17:20.992654 4551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-2/data/instance:
uuid: "05734e88c1794a1b82b177ab5c2c127f"
format_stamp: "Formatted at 2026-09-29 20:17:20 on dist-test-slave-4cx3"
server_key: "eae7ab8ba710e67e63ba26d4da728b5b"
server_key_iv: "3053135daf29066bb87854f713216178"
server_key_version: "encryptionkey@0"
I20260929 20:17:20.993512 4551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance:
uuid: "05734e88c1794a1b82b177ab5c2c127f"
format_stamp: "Formatted at 2026-09-29 20:17:20 on dist-test-slave-4cx3"
server_key: "eae7ab8ba710e67e63ba26d4da728b5b"
server_key_iv: "3053135daf29066bb87854f713216178"
server_key_version: "encryptionkey@0"
I20260929 20:17:20.998631 4551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 20:17:21.002545 4566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:21.004164 4551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:17:21.004349 4551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "05734e88c1794a1b82b177ab5c2c127f"
format_stamp: "Formatted at 2026-09-29 20:17:20 on dist-test-slave-4cx3"
server_key: "eae7ab8ba710e67e63ba26d4da728b5b"
server_key_iv: "3053135daf29066bb87854f713216178"
server_key_version: "encryptionkey@0"
I20260929 20:17:21.004583 4551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:21.039049 4551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:21.043668 4551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:21.043907 4551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:21.044813 4551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:17:21.046118 4551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:17:21.046191 4551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:21.046277 4551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:17:21.046320 4551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:21.080055 4551 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:46507
I20260929 20:17:21.080216 4678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:46507 every 8 connection(s)
I20260929 20:17:21.081485 4551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:17:21.089411 4412 heartbeater.cc:503] Master 127.27.185.126:39341 was elected leader, sending a full tablet report...
I20260929 20:17:21.090559 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 4551
I20260929 20:17:21.090763 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance
I20260929 20:17:21.091358 28389 external_mini_cluster.cc:1428] Setting key c0cd81a18d3acc5449900cfef058a171
Sep 29 20:17:21 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713040, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:17:21.102144 4414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:21.084063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:51349 (local address 127.27.185.126:39341)
0929 20:17:21.084244 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:21.084248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:21.085496 (+ 1248us) server_negotiation.cc:407] Connection header received
0929 20:17:21.087474 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:21.087483 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:21.087565 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:21.087677 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:21.089215 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:21.090278 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:21.092419 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:21.092832 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:21.092964 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:21.096402 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:21.096408 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:21.096413 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:21.096453 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:21.098854 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:21.099684 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:21.099691 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:21.099698 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:21.099829 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:21.100352 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:21.100357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:21.100360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:21.100665 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:21.100834 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:21.101581 (+ 747us) server_negotiation.cc:299] Negotiation successful
0929 20:17:21.101798 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:17:21.103070 4681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:21.084578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.67:51349)
0929 20:17:21.085287 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:21.085325 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:21.087129 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:21.087924 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:21.087932 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:21.088435 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:21.089018 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:21.089035 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:21.090427 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:21.090438 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:21.092273 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:21.092294 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:21.093060 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:21.093067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:21.093316 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:21.094000 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:21.094027 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:21.096191 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:21.098991 (+ 2800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:21.099000 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:21.099015 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:21.099573 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:21.099940 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:21.099945 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:21.099949 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:21.100231 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:21.100805 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:21.100813 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:21.101177 (+ 364us) client_negotiation.cc:769] Sending connection context
0929 20:17:21.101492 (+ 315us) client_negotiation.cc:240] Negotiation successful
0929 20:17:21.102007 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":292,"threads_started":1}
I20260929 20:17:21.104460 4679 heartbeater.cc:347] Connected to a master server at 127.27.185.126:39341
I20260929 20:17:21.104753 4679 heartbeater.cc:464] Registering TS with master...
I20260929 20:17:21.105566 4679 heartbeater.cc:511] Master 127.27.185.126:39341 requested a full tablet report, sending...
I20260929 20:17:21.106981 4223 ts_manager.cc:194] Registered new tserver with Master: 05734e88c1794a1b82b177ab5c2c127f (127.27.185.67:46507)
I20260929 20:17:21.107916 4223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.67@KRBTEST.COM'} at 127.27.185.67:51349
I20260929 20:17:21.118211 28389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.65.keytab.
Entry for principal oryx/127.27.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.65.keytab.
WARNING: no policy specified for oryx/127.27.185.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.185.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.66.keytab.
Entry for principal oryx/127.27.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.66.keytab.
WARNING: no policy specified for oryx/127.27.185.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.67.keytab.
Entry for principal oryx/127.27.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.67.keytab.
I20260929 20:17:21.258664 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 4284
Sep 29 20:17:21 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.65@KRBTEST.COM
I20260929 20:17:21.276006 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:21.260102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58090 (local address 127.27.185.65:36773)
0929 20:17:21.260817 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:21.260825 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:21.261321 (+ 496us) server_negotiation.cc:407] Connection header received
0929 20:17:21.261437 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:21.261444 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:21.261682 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:21.261825 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:21.263149 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:21.264284 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:21.265581 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:21.266147 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:21.266251 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:21.270294 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:21.270306 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:21.270321 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:21.270353 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:21.273106 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:21.273835 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:21.273843 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:21.273845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:21.273969 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:21.274339 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:21.274345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:21.274348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:21.274707 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:21.275034 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:21.275323 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 20:17:21.275638 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":408,"threads_started":1}
I20260929 20:17:21.277268 4407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:21.360561 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 4284
I20260929 20:17:21.377756 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 4418
Sep 29 20:17:21 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.66@KRBTEST.COM
I20260929 20:17:21.394734 4691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:21.378702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35794 (local address 127.27.185.66:36251)
0929 20:17:21.379311 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:21.379319 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:21.379345 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:17:21.379472 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:21.379480 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:21.379705 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:21.379840 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:21.381176 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:21.382249 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:21.383715 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:21.384260 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:21.384366 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:21.389008 (+ 4642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:21.389021 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:21.389035 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:21.389070 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:21.392345 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:21.392943 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:21.392949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:21.392952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:21.393065 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:21.393365 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:21.393368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:21.393370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:21.393706 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:21.393932 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:21.394193 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 20:17:21.394383 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":342,"threads_started":1}
I20260929 20:17:21.396389 4541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:21.479126 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 4418
I20260929 20:17:21.495672 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 4551
Sep 29 20:17:21 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.67@KRBTEST.COM
I20260929 20:17:21.512420 4693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:21.496695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45862 (local address 127.27.185.67:46507)
0929 20:17:21.497300 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:21.497307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:21.497334 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:17:21.497395 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:21.497401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:21.497606 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:21.497733 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:21.499063 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:21.500221 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:21.501466 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:21.502232 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:21.502328 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:21.506768 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:21.506779 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:21.506793 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:21.506828 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:21.509838 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:21.510416 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:21.510422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:21.510425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:21.510547 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:21.510882 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:21.510885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:21.510887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:21.511228 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:21.511555 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:21.511952 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 20:17:21.512145 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":311,"threads_started":1}
I20260929 20:17:21.513698 4674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:21.596869 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 4551
I20260929 20:17:21.612854 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 4191
I20260929 20:17:21.614010 4253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:21.717804 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 4191
I20260929 20:17:21.738356 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:39341
--webserver_interface=127.27.185.126
--webserver_port=37887
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:39341
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:17:22.066082 4696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:22.066396 4696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:22.066557 4696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:22.073894 4696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:17:22.074011 4696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:22.074100 4696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:22.074193 4696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:17:22.074282 4696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:17:22.084287 4696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:39341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:39341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=37887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:22.085758 4696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:22.087129 4696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:22.099306 4704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:22.099445 4701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:22.099330 4702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:22.100183 4696 server_base.cc:1034] running on GCE node
I20260929 20:17:22.101012 4696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:22.102437 4696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:22.103682 4696 hybrid_clock.cc:648] HybridClock initialized: now 1790713042103631 us; error 62 us; skew 500 ppm
I20260929 20:17:22.104058 4696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:17:22 dist-test-slave-4cx3 krb5kdc[4178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713042, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:17:22.110404 4696 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:17:22.112048 4696 webserver.cc:460] Webserver started at http://127.27.185.126:37887/ using document root <none> and password file <none>
I20260929 20:17:22.112816 4696 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:22.112982 4696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:22.118932 4696 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 20:17:22.121994 4711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:22.123531 4696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 20:17:22.123710 4696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "18ab230587d244f8a12fd275daadfbbe"
format_stamp: "Formatted at 2026-09-29 20:17:19 on dist-test-slave-4cx3"
server_key: "45692c7a8225dff1fc402e66bd9a274d"
server_key_iv: "e41d5716a342e426c496ac258609a9a8"
server_key_version: "encryptionkey@0"
I20260929 20:17:22.124765 4696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:22.140766 4696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:22.144996 4696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:22.145284 4696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:22.164208 4696 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:39341
I20260929 20:17:22.164227 4762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:39341 every 8 connection(s)
I20260929 20:17:22.165773 4696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:17:22.170012 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 4696
I20260929 20:17:22.170964 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:36773
--local_ip_for_outbound_sockets=127.27.185.65
--tserver_master_addrs=127.27.185.126:39341
--webserver_port=42245
--webserver_interface=127.27.185.65
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.185.65
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:17:22.174408 4763 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:17:22.179587 4763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe: Bootstrap starting.
I20260929 20:17:22.199131 4763 log.cc:824] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe: Log is configured to *not* fsync() on all Append() calls
I20260929 20:17:22.208389 4763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:17:22.208932 4763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe: Bootstrap complete.
I20260929 20:17:22.225193 4763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ab230587d244f8a12fd275daadfbbe" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 39341 } }
I20260929 20:17:22.227068 4763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18ab230587d244f8a12fd275daadfbbe, State: Initialized, Role: FOLLOWER
I20260929 20:17:22.228137 4763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ab230587d244f8a12fd275daadfbbe" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 39341 } }
I20260929 20:17:22.228485 4763 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:17:22.228667 4763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:17:22.229015 4763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [term 1 FOLLOWER]: Advancing to term 2
I20260929 20:17:22.235001 4763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ab230587d244f8a12fd275daadfbbe" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 39341 } }
I20260929 20:17:22.235651 4763 leader_election.cc:302] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18ab230587d244f8a12fd275daadfbbe; no voters:
I20260929 20:17:22.236315 4763 leader_election.cc:288] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 20:17:22.236482 4767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [term 2 FOLLOWER]: Leader election won for term 2
I20260929 20:17:22.237195 4767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [term 2 LEADER]: Becoming Leader. State: Replica: 18ab230587d244f8a12fd275daadfbbe, State: Running, Role: LEADER
I20260929 20:17:22.237968 4767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ab230587d244f8a12fd275daadfbbe" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 39341 } }
I20260929 20:17:22.245000 4767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18ab230587d244f8a12fd275daadfbbe. Latest consensus state: current_term: 2 leader_uuid: "18ab230587d244f8a12fd275daadfbbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ab230587d244f8a12fd275daadfbbe" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 39341 } } }
I20260929 20:17:22.245527 4767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [sys.catalog]: This master's current role is: LEADER
I20260929 20:17:22.245741 4769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "18ab230587d244f8a12fd275daadfbbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ab230587d244f8a12fd275daadfbbe" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 39341 } } }
I20260929 20:17:22.246117 4769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [sys.catalog]: This master's current role is: LEADER
I20260929 20:17:22.247750 4763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:17:22.247901 4772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:17:22.254829 4772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:17:22.256743 4772 catalog_manager.cc:1259] Loaded cluster ID: 2effacc342c74dffbc2e110be8b0c12a
I20260929 20:17:22.256959 4772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:17:22.262454 4772 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:17:22.264492 4772 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 18ab230587d244f8a12fd275daadfbbe: Loaded TSK: 0
I20260929 20:17:22.266480 4772 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 20:17:22.550652 4765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:22.550935 4765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:22.551097 4765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:22.558525 4765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:22.558645 4765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:22.558817 4765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:17:22.568753 4765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.65.keytab
--principal=oryx/127.27.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:36773
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=42245
--tserver_master_addrs=127.27.185.126:39341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:22.570196 4765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:22.571691 4765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:22.586236 4790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:22.587049 4791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:22.587311 4793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:22.587600 4765 server_base.cc:1034] running on GCE node
I20260929 20:17:22.588347 4765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:22.589424 4765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:22.590684 4765 hybrid_clock.cc:648] HybridClock initialized: now 1790713042590622 us; error 56 us; skew 500 ppm
I20260929 20:17:22.591048 4765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:17:22 dist-test-slave-4cx3 krb5kdc[4178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713042, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:17:22.594550 4765 init.cc:377] Logged in from keytab as oryx/127.27.185.65@KRBTEST.COM (short username oryx)
I20260929 20:17:22.596333 4765 webserver.cc:460] Webserver started at http://127.27.185.65:42245/ using document root <none> and password file <none>
I20260929 20:17:22.597054 4765 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:22.597193 4765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:22.602901 4765 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 20:17:22.605988 4800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:22.607534 4765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:17:22.607707 4765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "d0272ab77d9c45139a572b7672f61f75"
format_stamp: "Formatted at 2026-09-29 20:17:19 on dist-test-slave-4cx3"
server_key: "3f8068ff83988974947db94b021d5e6d"
server_key_iv: "7f1e1f56f4ba7c27cc45cc093443b5cf"
server_key_version: "encryptionkey@0"
I20260929 20:17:22.608691 4765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:22.629405 4765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:22.633613 4765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:22.633898 4765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:22.634896 4765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:17:22.636613 4765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:17:22.636731 4765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:22.636864 4765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:17:22.636953 4765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:22.671126 4765 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:36773
I20260929 20:17:22.671173 4912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:36773 every 8 connection(s)
I20260929 20:17:22.672637 4765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:17:22.676927 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 4765
I20260929 20:17:22.677495 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:36251
--local_ip_for_outbound_sockets=127.27.185.66
--tserver_master_addrs=127.27.185.126:39341
--webserver_port=33015
--webserver_interface=127.27.185.66
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.185.66
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:17:22 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:22 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:22.697077 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:22.675729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.65:56369)
0929 20:17:22.679706 (+ 3977us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:22.679752 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:22.681577 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:22.683299 (+ 1722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:22.683307 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:22.684157 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:22.685029 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:22.685056 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:22.686764 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:22.686773 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:22.689067 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:22.689085 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:22.690213 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:22.690222 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:22.690535 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:22.691300 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:22.691334 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:22.695663 (+ 4329us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3727,"thread_start_us":374,"threads_started":1}
W20260929 20:17:22.697100 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:22.675690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:56369 (local address 127.27.185.126:39341)
0929 20:17:22.676617 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:22.676637 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:22.679903 (+ 3266us) server_negotiation.cc:407] Connection header received
0929 20:17:22.682249 (+ 2346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:22.682271 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:22.682762 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:22.683177 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:22.685268 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:22.686580 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:22.689227 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:22.689937 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:22.690108 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:22.696041 (+ 5933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.65:56369: BlockingRecv error: recv got EOF from 127.27.185.65:56369 (error 108)
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":349,"threads_started":1}
W20260929 20:17:22.698066 4913 heartbeater.cc:644] Failed to heartbeat to 127.27.185.126:39341 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.185.126:39341: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Sep 29 20:17:22 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:22 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:22.707247 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:22.698755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.65:45211)
0929 20:17:22.698950 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:22.698965 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:22.699049 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:22.699500 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:22.699504 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:22.699794 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:22.699952 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:22.699960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:22.701523 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:22.701527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:22.702391 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:22.702402 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:22.703318 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:22.703322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:22.703529 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:22.704211 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:22.704229 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:22.707055 (+ 2826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 20:17:22.707535 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:22.698855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:45211 (local address 127.27.185.126:39341)
0929 20:17:22.699067 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:22.699075 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:22.699119 (+ 44us) server_negotiation.cc:407] Connection header received
0929 20:17:22.699188 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:22.699194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:22.699285 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:22.699411 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:22.700097 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:22.701370 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:22.702531 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:22.703184 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:22.703305 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:22.707291 (+ 3986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.65:45211: BlockingRecv error: recv got EOF from 127.27.185.65:45211 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 29 20:17:22 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:22 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:22.716017 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:22.708056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.65:40875)
0929 20:17:22.708219 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:22.708244 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:22.708352 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:22.708707 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:22.708711 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:22.709007 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:22.709139 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:22.709147 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:22.710201 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:22.710207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:22.711131 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:22.711142 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:22.711984 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:22.711990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:22.712204 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:22.712815 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:22.712832 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:22.715703 (+ 2871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 20:17:22.716323 4913 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 20:17:22.716545 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:22.708231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:40875 (local address 127.27.185.126:39341)
0929 20:17:22.708438 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:22.708445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:22.708461 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:17:22.708511 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:22.708514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:22.708580 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:22.708682 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:22.709275 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:22.710060 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:22.711269 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:22.711840 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:22.711975 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:22.715920 (+ 3945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.65:40875: BlockingRecv error: recv got EOF from 127.27.185.65:40875 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260929 20:17:23.014932 4917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:23.015283 4917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:23.015507 4917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:23.022935 4917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:23.023087 4917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:23.023274 4917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:17:23.033293 4917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.66.keytab
--principal=oryx/127.27.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:36251
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=33015
--tserver_master_addrs=127.27.185.126:39341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:23.034677 4917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:23.036048 4917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:23.049320 4922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:23.049227 4923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:23.049260 4925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:23.050140 4917 server_base.cc:1034] running on GCE node
I20260929 20:17:23.050735 4917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:23.051697 4917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:23.053012 4917 hybrid_clock.cc:648] HybridClock initialized: now 1790713043052938 us; error 56 us; skew 500 ppm
I20260929 20:17:23.053381 4917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:17:23 dist-test-slave-4cx3 krb5kdc[4178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713043, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:17:23.056727 4917 init.cc:377] Logged in from keytab as oryx/127.27.185.66@KRBTEST.COM (short username oryx)
I20260929 20:17:23.058194 4917 webserver.cc:460] Webserver started at http://127.27.185.66:33015/ using document root <none> and password file <none>
I20260929 20:17:23.058859 4917 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:23.058974 4917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:23.064333 4917 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:17:23.066886 4932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:23.068323 4917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:17:23.068511 4917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "3af90c3ef7c54392b87acb8049373bb7"
format_stamp: "Formatted at 2026-09-29 20:17:20 on dist-test-slave-4cx3"
server_key: "c429aea971dfaa0e4e383243c69be6c1"
server_key_iv: "6ee09048f716c736270d11b8da02309a"
server_key_version: "encryptionkey@0"
I20260929 20:17:23.069535 4917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:23.095016 4917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:23.099694 4917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:23.099926 4917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:23.100894 4917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:17:23.102185 4917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:17:23.102268 4917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:23.102339 4917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:17:23.102394 4917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:23.136442 4917 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:36251
I20260929 20:17:23.136515 5044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:36251 every 8 connection(s)
I20260929 20:17:23.137965 4917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:17:23.143252 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 4917
I20260929 20:17:23.144025 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:46507
--local_ip_for_outbound_sockets=127.27.185.67
--tserver_master_addrs=127.27.185.126:39341
--webserver_port=46185
--webserver_interface=127.27.185.67
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.185.67
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:17:23 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:23 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 20:17:23.156680 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:23.140733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:51973 (local address 127.27.185.126:39341)
0929 20:17:23.140957 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:23.140964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:23.142082 (+ 1118us) server_negotiation.cc:407] Connection header received
0929 20:17:23.144566 (+ 2484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:23.144571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:23.144639 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:23.144726 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:23.146931 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:23.147725 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:23.150319 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:23.150766 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:23.150862 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:23.156416 (+ 5554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.66:51973: BlockingRecv error: recv got EOF from 127.27.185.66:51973 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 20:17:23.157397 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:23.141201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.66:51973)
0929 20:17:23.141913 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:23.141957 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:23.144242 (+ 2285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:23.145038 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:23.145054 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:23.145875 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:23.146715 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:23.146743 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:23.147914 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:23.147923 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:23.150170 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:23.150188 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:23.151008 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:23.151018 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:23.151364 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:23.152156 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:23.152179 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:23.156154 (+ 3975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":277,"threads_started":1}
W20260929 20:17:23.158087 5045 heartbeater.cc:644] Failed to heartbeat to 127.27.185.126:39341 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.185.126:39341: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Sep 29 20:17:23 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:23 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:23.168334 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:23.158735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.66:53819)
0929 20:17:23.158929 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:23.158951 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:23.159052 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:23.159371 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:23.159374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:23.159777 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:23.159920 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:23.159929 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:23.161428 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:23.161431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:23.162303 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:23.162313 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:23.163524 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:23.163531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:23.163701 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:23.164408 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:23.164433 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:23.168046 (+ 3613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 20:17:23.168651 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:23.158851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:53819 (local address 127.27.185.126:39341)
0929 20:17:23.159042 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:23.159050 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:23.159072 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:17:23.159137 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:23.159145 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:23.159231 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:23.159348 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:23.160068 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:23.161288 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:23.162447 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:23.163260 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:23.163486 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:23.168350 (+ 4864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.66:53819: BlockingRecv error: recv got EOF from 127.27.185.66:53819 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 20:17:23 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:23 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:23.178920 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:23.169255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.66:54665)
0929 20:17:23.169535 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:23.169553 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:23.169645 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:23.170003 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:23.170008 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:23.170373 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:23.170551 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:23.170562 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:23.172019 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:23.172024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:23.173546 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:23.173560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:23.174641 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:23.174647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:23.174827 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:23.175484 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:23.175507 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:23.178706 (+ 3199us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171}
W20260929 20:17:23.179168 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:23.169393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:54665 (local address 127.27.185.126:39341)
0929 20:17:23.169570 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:23.169575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:23.169596 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:17:23.169768 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:23.169774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:23.169854 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:23.169956 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:23.170712 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:23.171865 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:23.173791 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:23.174496 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:23.174613 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:23.178947 (+ 4334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.66:54665: BlockingRecv error: recv got EOF from 127.27.185.66:54665 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 20:17:23.179307 5045 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 20:17:23.478034 5048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:23.478369 5048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:23.478550 5048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:23.486018 5048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:23.486179 5048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:23.486366 5048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:17:23.496701 5048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/krb5kdc/oryx_127.27.185.67.keytab
--principal=oryx/127.27.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:46507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=46185
--tserver_master_addrs=127.27.185.126:39341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:23.498155 5048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:23.499616 5048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:23.513023 5056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:23.513041 5054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:23.513283 5053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:23.513835 5048 server_base.cc:1034] running on GCE node
I20260929 20:17:23.514514 5048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:23.515534 5048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:23.516757 5048 hybrid_clock.cc:648] HybridClock initialized: now 1790713043516706 us; error 47 us; skew 500 ppm
I20260929 20:17:23.517083 5048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:17:23 dist-test-slave-4cx3 krb5kdc[4178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713043, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:17:23.520274 5048 init.cc:377] Logged in from keytab as oryx/127.27.185.67@KRBTEST.COM (short username oryx)
I20260929 20:17:23.521793 5048 webserver.cc:460] Webserver started at http://127.27.185.67:46185/ using document root <none> and password file <none>
I20260929 20:17:23.522509 5048 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:23.522667 5048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:23.528141 5048 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260929 20:17:23.530952 5063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:23.532403 5048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:17:23.532588 5048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "05734e88c1794a1b82b177ab5c2c127f"
format_stamp: "Formatted at 2026-09-29 20:17:20 on dist-test-slave-4cx3"
server_key: "eae7ab8ba710e67e63ba26d4da728b5b"
server_key_iv: "3053135daf29066bb87854f713216178"
server_key_version: "encryptionkey@0"
I20260929 20:17:23.533556 5048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:23.546368 5048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:23.550688 5048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:23.550966 5048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:23.551990 5048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:17:23.553278 5048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:17:23.553402 5048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:23.553544 5048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:17:23.553634 5048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:23.587872 5048 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:46507
I20260929 20:17:23.587958 5175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:46507 every 8 connection(s)
I20260929 20:17:23.589349 5048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:17:23.589821 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 5048
Sep 29 20:17:23 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:23 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 20:17:23.605983 5179 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:23.592088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:41835 (local address 127.27.185.126:39341)
0929 20:17:23.592532 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:23.592536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:23.593795 (+ 1259us) server_negotiation.cc:407] Connection header received
0929 20:17:23.596219 (+ 2424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:23.596223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:23.596302 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:23.596420 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:23.597951 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:23.598759 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:23.600337 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:23.600792 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:23.600940 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:23.605656 (+ 4716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.67:41835: BlockingRecv error: recv got EOF from 127.27.185.67:41835 (error 108)
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":262,"threads_started":1}
I20260929 20:17:23.606398 4915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:23.591336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60340 (local address 127.27.185.126:39341)
0929 20:17:23.591597 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:23.591604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:23.591626 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:17:23.591797 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:23.591804 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:23.591890 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:23.592000 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:23.594070 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:23.595191 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:23.596583 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:23.597017 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:23.597165 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:23.599915 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:23.599923 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:23.599939 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:23.599989 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:23.603017 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:23.603787 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:23.603793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:23.603799 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:23.603964 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:23.604362 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:23.604366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:23.604368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:23.604789 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:23.605125 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:23.605537 (+ 412us) server_negotiation.cc:299] Negotiation successful
0929 20:17:23.605913 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260929 20:17:23.606843 5180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:23.592794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.67:41835)
0929 20:17:23.593640 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:23.593679 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:23.595958 (+ 2279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:23.596684 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:23.596693 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:23.597201 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:23.597780 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:23.597797 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:23.598915 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:23.598919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:23.600190 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:23.600202 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:23.601028 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:23.601037 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:23.601284 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:23.601990 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:23.602011 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:23.605441 (+ 3430us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":329,"threads_started":1}
W20260929 20:17:23.607513 5176 heartbeater.cc:644] Failed to heartbeat to 127.27.185.126:39341 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.185.126:39341: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Sep 29 20:17:23 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:23 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:23.615581 5180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:23.608033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.67:40985)
0929 20:17:23.608215 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:23.608233 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:23.608333 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:23.608636 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:23.608641 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:23.609060 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:23.609256 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:23.609268 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:23.610314 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:23.610318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:23.611175 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:23.611185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:23.611987 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:23.611991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:23.612108 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:23.612687 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:23.612704 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:23.615280 (+ 2576us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 20:17:23.616044 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:23.608155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:40985 (local address 127.27.185.126:39341)
0929 20:17:23.608295 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:23.608298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:23.608314 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:17:23.608434 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:23.608437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:23.608508 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:23.608610 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:23.609428 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:23.610155 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:23.611393 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:23.611852 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:23.611949 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:23.615516 (+ 3567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.67:40985: BlockingRecv error: recv got EOF from 127.27.185.67:40985 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 29 20:17:23 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:23 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:23.623715 5180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:23.616239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.67:32799)
0929 20:17:23.616440 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:23.616460 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:23.616555 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:23.616886 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:23.616891 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:23.617193 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:23.617331 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:23.617339 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:23.618302 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:23.618307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:23.619184 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:23.619195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:23.619953 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:23.619958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:23.620123 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:23.620674 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:23.620696 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:23.623368 (+ 2672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 20:17:23.623963 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:23.616295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:32799 (local address 127.27.185.126:39341)
0929 20:17:23.616532 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:23.616537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:23.616554 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:17:23.616687 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:23.616692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:23.616753 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:23.616866 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:23.617470 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:23.618165 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:23.619328 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:23.619806 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:23.619944 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:23.623737 (+ 3793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.67:32799: BlockingRecv error: recv got EOF from 127.27.185.67:32799 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
W20260929 20:17:23.624038 5176 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 20:17:23 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:23 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:23.725843 5181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:23.716985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.65:53589)
0929 20:17:23.717527 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:23.717543 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:23.717657 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:23.717977 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:23.717981 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:23.718433 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:23.718679 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:23.718692 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:23.720115 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:23.720121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:23.721165 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:23.721177 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:23.721890 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:23.721894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:23.722021 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:23.722655 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:23.722674 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:23.725599 (+ 2925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":301,"threads_started":1}
W20260929 20:17:23.725999 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:23.717051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:53589 (local address 127.27.185.126:39341)
0929 20:17:23.717239 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:23.717243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:23.717625 (+ 382us) server_negotiation.cc:407] Connection header received
0929 20:17:23.717758 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:23.717762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:23.717829 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:23.717940 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:23.718857 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:23.719947 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:23.721298 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:23.721732 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:23.721830 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:23.725800 (+ 3970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.65:53589: BlockingRecv error: recv got EOF from 127.27.185.65:53589 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 29 20:17:24 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:24 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:24.188814 5182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:24.180021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.66:36607)
0929 20:17:24.180526 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:24.180577 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:24.180727 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:24.181176 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:24.181179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:24.181828 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:24.182084 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:24.182095 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:24.183292 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:24.183296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:24.184306 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:24.184319 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:24.185017 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:24.185021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:24.185143 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:24.185732 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:24.185749 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:24.188599 (+ 2850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":264,"threads_started":1}
W20260929 20:17:24.189031 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:24.180094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:36607 (local address 127.27.185.126:39341)
0929 20:17:24.180342 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:24.180350 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:24.180650 (+ 300us) server_negotiation.cc:407] Connection header received
0929 20:17:24.180896 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:24.180901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:24.180997 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:24.181098 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:24.182236 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:24.183162 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:24.184435 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:24.184863 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:24.184963 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:24.188787 (+ 3824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.66:36607: BlockingRecv error: recv got EOF from 127.27.185.66:36607 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 20:17:24 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:24 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:24.633581 5183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:24.624651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.67:53299)
0929 20:17:24.625181 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:24.625197 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:24.625310 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:24.625644 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:24.625647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:24.626116 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:24.626303 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:24.626311 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:24.627330 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:24.627334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:24.628831 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:24.628845 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:24.629746 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:24.629750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:24.629869 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:24.630506 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:24.630524 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:24.633277 (+ 2753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":301,"threads_started":1}
W20260929 20:17:24.633670 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:24.624762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:53299 (local address 127.27.185.126:39341)
0929 20:17:24.624938 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:24.624942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:24.625272 (+ 330us) server_negotiation.cc:407] Connection header received
0929 20:17:24.625451 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:24.625455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:24.625516 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:24.625601 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:24.626454 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:24.627187 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:24.628985 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:24.629592 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:24.629687 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:24.633469 (+ 3782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.67:53299: BlockingRecv error: recv got EOF from 127.27.185.67:53299 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 20:17:24 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:24 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:24.735198 5184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:24.726820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.65:54703)
0929 20:17:24.727316 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:24.727332 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:24.727471 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:24.727830 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:24.727833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:24.728281 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:24.728472 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:24.728483 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:24.729630 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:24.729634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:24.730586 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:24.730598 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:24.731310 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:24.731314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:24.731505 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:24.732129 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:24.732146 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:24.734946 (+ 2800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":276,"threads_started":1}
W20260929 20:17:24.735584 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:24.726917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:54703 (local address 127.27.185.126:39341)
0929 20:17:24.727172 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:24.727178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:24.727566 (+ 388us) server_negotiation.cc:407] Connection header received
0929 20:17:24.727636 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:24.727639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:24.727711 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:24.727791 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:24.728651 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:24.729477 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:24.730762 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:24.731176 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:24.731275 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:24.735234 (+ 3959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.65:54703: BlockingRecv error: recv got EOF from 127.27.185.65:54703 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 29 20:17:25 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:25 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:25.198458 5185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:25.189849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.66:60457)
0929 20:17:25.190345 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:25.190362 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:25.190487 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:25.190874 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:25.190878 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:25.191274 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:25.191521 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:25.191531 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:25.192742 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:25.192748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:25.194063 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:25.194075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:25.194754 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:25.194758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:25.194866 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:25.195486 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:25.195515 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:25.198210 (+ 2695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":284,"threads_started":1}
W20260929 20:17:25.198642 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:25.189926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:60457 (local address 127.27.185.126:39341)
0929 20:17:25.190156 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:25.190163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:25.190450 (+ 287us) server_negotiation.cc:407] Connection header received
0929 20:17:25.190662 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:25.190666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:25.190743 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:25.190838 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:25.191669 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:25.192574 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:25.194188 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:25.194608 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:25.194699 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:25.198399 (+ 3700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.66:60457: BlockingRecv error: recv got EOF from 127.27.185.66:60457 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 20:17:25 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:25 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:25.643455 5186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:25.634635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.67:53597)
0929 20:17:25.635184 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:25.635205 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:25.635355 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:25.635792 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:25.635797 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:25.636241 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:25.636425 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:25.636436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:25.637697 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:25.637704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:25.638730 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:25.638742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:25.639507 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:25.639513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:25.639686 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:25.640234 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:25.640252 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:25.643058 (+ 2806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":261,"threads_started":1}
W20260929 20:17:25.643543 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:25.634834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:53597 (local address 127.27.185.126:39341)
0929 20:17:25.635056 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:25.635064 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:25.635320 (+ 256us) server_negotiation.cc:407] Connection header received
0929 20:17:25.635548 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:25.635553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:25.635636 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:25.635754 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:25.636625 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:25.637548 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:25.638897 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:25.639310 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:25.639500 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:25.643277 (+ 3777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.67:53597: BlockingRecv error: recv got EOF from 127.27.185.67:53597 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 29 20:17:25 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:25 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:25.744781 5187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:25.736246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.65:37667)
0929 20:17:25.736775 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:25.736812 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:25.736998 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:25.737386 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:25.737389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:25.737730 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:25.737957 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:25.737973 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:25.739101 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:25.739105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:25.740092 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:25.740103 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:25.740816 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:25.740820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:25.740960 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:25.741536 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:25.741554 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:25.744395 (+ 2841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":293,"threads_started":1}
W20260929 20:17:25.744892 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:25.736331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:37667 (local address 127.27.185.126:39341)
0929 20:17:25.736636 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:25.736643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:25.736899 (+ 256us) server_negotiation.cc:407] Connection header received
0929 20:17:25.737122 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:25.737127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:25.737199 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:25.737338 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:25.738111 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:25.738970 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:25.740250 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:25.740691 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:25.740787 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:25.744610 (+ 3823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.65:37667: BlockingRecv error: recv got EOF from 127.27.185.65:37667 (error 108)
Metrics: {"server-negotiator.queue_time_us":144}
Sep 29 20:17:26 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:26 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:26.208834 5188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:26.199587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.66:53419)
0929 20:17:26.200145 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:26.200165 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:26.200333 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:26.200831 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:26.200835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:26.201165 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:26.201367 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:26.201378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:26.202966 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:26.202974 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:26.204136 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:26.204149 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:26.204899 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:26.204914 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:26.205099 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:26.205705 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:26.205722 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:26.208473 (+ 2751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":410,"threads_started":1}
W20260929 20:17:26.208899 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:26.199652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:53419 (local address 127.27.185.126:39341)
0929 20:17:26.199997 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:26.200005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:26.200266 (+ 261us) server_negotiation.cc:407] Connection header received
0929 20:17:26.200534 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:26.200538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:26.200607 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:26.200785 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:26.201629 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:26.202759 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:26.204313 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:26.204743 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:26.204876 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:26.208666 (+ 3790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.66:53419: BlockingRecv error: recv got EOF from 127.27.185.66:53419 (error 108)
Metrics: {"server-negotiator.queue_time_us":163}
Sep 29 20:17:26 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:26 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:26.652786 5189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:26.644305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.67:58107)
0929 20:17:26.644843 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:26.644860 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:26.644975 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:26.645259 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:26.645262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:26.645648 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:26.645810 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:26.645821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:26.646872 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:26.646877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:26.647957 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:26.647971 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:26.648713 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:26.648717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:26.648849 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:26.649497 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:26.649516 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:26.652538 (+ 3022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":308,"threads_started":1}
W20260929 20:17:26.653010 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:26.644439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:58107 (local address 127.27.185.126:39341)
0929 20:17:26.644634 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:26.644639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:26.644995 (+ 356us) server_negotiation.cc:407] Connection header received
0929 20:17:26.645063 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:26.645066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:26.645127 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:26.645209 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:26.645965 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:26.646718 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:26.648123 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:26.648575 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:26.648672 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:26.652785 (+ 4113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.67:58107: BlockingRecv error: recv got EOF from 127.27.185.67:58107 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 20:17:26 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:26 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:26.754382 5190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:26.745738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.65:47271)
0929 20:17:26.746181 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:26.746197 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:26.746313 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:26.746654 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:26.746658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:26.747054 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:26.747223 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:26.747232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:26.748346 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:26.748350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:26.749430 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:26.749443 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:26.750131 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:26.750135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:26.750302 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:26.750874 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:26.750892 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:26.754050 (+ 3158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":248,"threads_started":1}
W20260929 20:17:26.754623 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:26.745762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:47271 (local address 127.27.185.126:39341)
0929 20:17:26.745995 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:26.746005 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:26.746275 (+ 270us) server_negotiation.cc:407] Connection header received
0929 20:17:26.746425 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:26.746430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:26.746496 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:26.746578 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:26.747363 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:26.748219 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:26.749556 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:26.749978 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:26.750072 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:26.754274 (+ 4202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.65:47271: BlockingRecv error: recv got EOF from 127.27.185.65:47271 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 20:17:27 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:27 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:27.218585 5191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:27.209844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.66:60729)
0929 20:17:27.210386 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:27.210402 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:27.210523 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:27.210825 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:27.210828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:27.211203 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:27.211352 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:27.211361 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:27.212453 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:27.212457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:27.213693 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:27.213708 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:27.214584 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:27.214598 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:27.214742 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:27.215474 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:27.215494 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:27.218273 (+ 2779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":316,"threads_started":1}
W20260929 20:17:27.218916 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:27.209982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:60729 (local address 127.27.185.126:39341)
0929 20:17:27.210160 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:27.210165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:27.210477 (+ 312us) server_negotiation.cc:407] Connection header received
0929 20:17:27.210626 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:27.210630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:27.210695 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:27.210778 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:27.211497 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:27.212296 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:27.213841 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:27.214299 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:27.214473 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:27.218633 (+ 4160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.66:60729: BlockingRecv error: recv got EOF from 127.27.185.66:60729 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 20:17:27 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:27 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:27.666391 5192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:27.653758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.67:51885)
0929 20:17:27.654274 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:27.654291 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:27.654412 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:27.654749 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:27.654752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:27.655266 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:27.655489 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:27.655500 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:27.656531 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:27.656535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:27.657486 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:27.657497 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:27.658231 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:27.658235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:27.658355 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:27.663175 (+ 4820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:27.663196 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:27.666144 (+ 2948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":291,"threads_started":1}
W20260929 20:17:27.666751 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:27.653880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:51885 (local address 127.27.185.126:39341)
0929 20:17:27.654068 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:27.654073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:27.654367 (+ 294us) server_negotiation.cc:407] Connection header received
0929 20:17:27.654527 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:27.654531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:27.654613 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:27.654705 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:27.655624 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:27.656401 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:27.657628 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:27.658042 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:27.658206 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:27.666472 (+ 8266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.67:51885: BlockingRecv error: recv got EOF from 127.27.185.67:51885 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 20:17:27 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:27 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:27.764343 5193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:27.755479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.65:44023)
0929 20:17:27.756008 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:27.756026 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:27.756184 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:27.756541 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:27.756546 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:27.756998 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:27.757220 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:27.757234 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:27.758701 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:27.758707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:27.759847 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:27.759860 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:27.760513 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:27.760518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:27.760644 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:27.761237 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:27.761254 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:27.764089 (+ 2835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":262,"threads_started":1}
W20260929 20:17:27.764523 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:27.755553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:44023 (local address 127.27.185.126:39341)
0929 20:17:27.755788 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:27.755796 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:27.756120 (+ 324us) server_negotiation.cc:407] Connection header received
0929 20:17:27.756310 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:27.756315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:27.756394 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:27.756505 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:27.757382 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:27.758516 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:27.759979 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:27.760378 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:27.760478 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:27.764309 (+ 3831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.65:44023: BlockingRecv error: recv got EOF from 127.27.185.65:44023 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 20:17:28 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:28 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:28.228534 5194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:28.219626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.66:34713)
0929 20:17:28.220208 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:28.220224 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:28.220400 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:28.220772 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:28.220778 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:28.221218 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:28.221441 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:28.221451 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:28.222565 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:28.222572 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:28.223577 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:28.223590 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:28.224329 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:28.224344 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:28.224508 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:28.225101 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:28.225119 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:28.228105 (+ 2986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":268,"threads_started":1}
W20260929 20:17:28.228585 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:28.219880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:34713 (local address 127.27.185.126:39341)
0929 20:17:28.220118 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:28.220125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:28.220321 (+ 196us) server_negotiation.cc:407] Connection header received
0929 20:17:28.220541 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:28.220547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:28.220623 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:28.220733 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:28.221600 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:28.222412 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:28.223721 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:28.224147 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:28.224280 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:28.228349 (+ 4069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.66:34713: BlockingRecv error: recv got EOF from 127.27.185.66:34713 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 29 20:17:28 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:28 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:28.676488 5195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:28.667358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.67:35585)
0929 20:17:28.667986 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:28.668003 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:28.668121 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:28.668443 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:28.668446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:28.668818 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:28.669001 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:28.669010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:28.670061 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:28.670065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:28.670977 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:28.670989 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:28.672003 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:28.672007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:28.672133 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:28.672833 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:28.672850 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:28.676121 (+ 3271us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":288,"threads_started":1}
W20260929 20:17:28.676702 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:28.667514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:35585 (local address 127.27.185.126:39341)
0929 20:17:28.667686 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:28.667691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:28.668120 (+ 429us) server_negotiation.cc:407] Connection header received
0929 20:17:28.668233 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:28.668236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:28.668302 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:28.668399 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:28.669148 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:28.669904 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:28.671137 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:28.671863 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:28.671988 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:28.676391 (+ 4403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.67:35585: BlockingRecv error: recv got EOF from 127.27.185.67:35585 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 20:17:28 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:28 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:28.773906 5196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:28.765447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.65:43299)
0929 20:17:28.765974 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:28.765991 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:28.766118 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:28.766484 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:28.766488 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:28.766878 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:28.767040 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:28.767049 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:28.768040 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:28.768045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:28.768959 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:28.768971 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:28.769733 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:28.769737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:28.769875 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:28.770493 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:28.770511 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:28.773508 (+ 2997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":309,"threads_started":1}
W20260929 20:17:28.774055 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:28.765626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:43299 (local address 127.27.185.126:39341)
0929 20:17:28.765881 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:28.765887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:28.766078 (+ 191us) server_negotiation.cc:407] Connection header received
0929 20:17:28.766224 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:28.766228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:28.766292 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:28.766374 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:28.767164 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:28.767916 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:28.769089 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:28.769560 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:28.769654 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:28.773762 (+ 4108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.65:43299: BlockingRecv error: recv got EOF from 127.27.185.65:43299 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 29 20:17:29 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:29 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:29.238258 5197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:29.229563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.66:40273)
0929 20:17:29.230077 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:29.230094 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:29.230248 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:29.230614 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:29.230619 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:29.231045 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:29.231194 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:29.231203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:29.232258 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:29.232264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:29.233185 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:29.233198 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:29.233953 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:29.233959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:29.234148 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:29.234856 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:29.234882 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:29.238050 (+ 3168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":297,"threads_started":1}
W20260929 20:17:29.238592 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:29.229566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:40273 (local address 127.27.185.126:39341)
0929 20:17:29.229758 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:29.229766 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:29.230187 (+ 421us) server_negotiation.cc:407] Connection header received
0929 20:17:29.230392 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:29.230397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:29.230469 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:29.230579 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:29.231360 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:29.232117 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:29.233339 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:29.233792 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:29.233934 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:29.238277 (+ 4343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.66:40273: BlockingRecv error: recv got EOF from 127.27.185.66:40273 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 20:17:29 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:29 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:29.686374 5198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:29.677572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.67:49803)
0929 20:17:29.678162 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:29.678180 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:29.678354 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:29.678714 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:29.678718 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:29.679220 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:29.679423 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:29.679434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:29.680456 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:29.680463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:29.681447 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:29.681459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:29.682271 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:29.682277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:29.682458 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:29.683126 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:29.683145 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:29.686157 (+ 3012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":343,"threads_started":1}
W20260929 20:17:29.686617 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:29.677632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:49803 (local address 127.27.185.126:39341)
0929 20:17:29.677961 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:29.677970 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:29.678282 (+ 312us) server_negotiation.cc:407] Connection header received
0929 20:17:29.678506 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:29.678511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:29.678574 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:29.678680 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:29.679579 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:29.680307 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:29.681593 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:29.682114 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:29.682255 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:29.686348 (+ 4093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.67:49803: BlockingRecv error: recv got EOF from 127.27.185.67:49803 (error 108)
Metrics: {"server-negotiator.queue_time_us":147}
Sep 29 20:17:29 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:29 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:29.784185 5199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:29.775049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.65:52517)
0929 20:17:29.775667 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:29.775684 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:29.775798 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:29.776123 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:29.776127 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:29.776546 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:29.776712 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:29.776721 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:29.777818 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:29.777822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:29.778786 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:29.778798 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:29.779633 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:29.779638 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:29.779762 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:29.780361 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:29.780378 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:29.783935 (+ 3557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":326,"threads_started":1}
W20260929 20:17:29.784538 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:29.775220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:52517 (local address 127.27.185.126:39341)
0929 20:17:29.775460 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:29.775465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:29.775766 (+ 301us) server_negotiation.cc:407] Connection header received
0929 20:17:29.775912 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:29.775916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:29.775982 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:29.776074 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:29.776868 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:29.777649 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:29.778949 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:29.779472 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:29.779584 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:29.784160 (+ 4576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.65:52517: BlockingRecv error: recv got EOF from 127.27.185.65:52517 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 20:17:30 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:30 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:30.247560 5200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:30.239250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.66:39627)
0929 20:17:30.239878 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:30.239894 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:30.240021 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:30.240312 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:30.240315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:30.240707 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:30.240868 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:30.240877 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:30.241834 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:30.241838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:30.242745 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:30.242758 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:30.243454 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:30.243463 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:30.243587 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:30.244211 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:30.244229 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:30.247213 (+ 2984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":321,"threads_started":1}
W20260929 20:17:30.247720 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:30.239359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:39627 (local address 127.27.185.126:39341)
0929 20:17:30.239576 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:30.239580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:30.239979 (+ 399us) server_negotiation.cc:407] Connection header received
0929 20:17:30.240122 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:30.240126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:30.240187 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:30.240269 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:30.240998 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:30.241716 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:30.242874 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:30.243251 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:30.243341 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:30.247469 (+ 4128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.66:39627: BlockingRecv error: recv got EOF from 127.27.185.66:39627 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 29 20:17:30 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:30 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:30.696403 5201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:30.687290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.67:49543)
0929 20:17:30.687885 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:30.687920 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:30.688038 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:30.688349 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:30.688353 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:30.688767 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:30.688950 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:30.688958 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:30.689944 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:30.689948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:30.691461 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:30.691480 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:30.692344 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:30.692348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:30.692507 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:30.693130 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:30.693147 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:30.696061 (+ 2914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":236,"threads_started":1}
W20260929 20:17:30.696498 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:30.687412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:49543 (local address 127.27.185.126:39341)
0929 20:17:30.687637 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:30.687647 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:30.688001 (+ 354us) server_negotiation.cc:407] Connection header received
0929 20:17:30.688142 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:30.688146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:30.688216 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:30.688307 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:30.689080 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:30.689805 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:30.691613 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:30.692167 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:30.692310 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:30.696256 (+ 3946us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.67:49543: BlockingRecv error: recv got EOF from 127.27.185.67:49543 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 20:17:30 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:30 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:30.794967 5202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:30.785205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.65:45535)
0929 20:17:30.785845 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:30.785862 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:30.785982 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:30.786289 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:30.786293 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:30.786738 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:30.786904 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:30.786916 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:30.787957 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:30.787961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:30.789230 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:30.789247 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:30.790276 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:30.790281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:30.790394 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:30.791021 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:30.791039 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:30.794528 (+ 3489us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":361,"threads_started":1}
W20260929 20:17:30.795020 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:30.785360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:45535 (local address 127.27.185.126:39341)
0929 20:17:30.785548 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:30.785553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:30.785946 (+ 393us) server_negotiation.cc:407] Connection header received
0929 20:17:30.786086 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:30.786090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:30.786152 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:30.786245 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:30.787044 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:30.787816 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:30.789433 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:30.790107 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:30.790200 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:30.794763 (+ 4563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.65:45535: BlockingRecv error: recv got EOF from 127.27.185.65:45535 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 20:17:31 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:31 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:31.257740 5203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:31.248591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.66:52425)
0929 20:17:31.249261 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:31.249279 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:31.249467 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:31.249959 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:31.249965 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:31.250509 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:31.250739 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:31.250753 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:31.252074 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:31.252078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:31.252992 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:31.253004 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:31.253848 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:31.253853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:31.253974 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:31.254605 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:31.254624 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:31.257518 (+ 2894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":354,"threads_started":1}
W20260929 20:17:31.257952 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:31.248796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:52425 (local address 127.27.185.126:39341)
0929 20:17:31.249079 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:31.249086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:31.249393 (+ 307us) server_negotiation.cc:407] Connection header received
0929 20:17:31.249622 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:31.249628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:31.249753 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:31.249860 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:31.250894 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:31.251927 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:31.253156 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:31.253631 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:31.253743 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:31.257730 (+ 3987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.66:52425: BlockingRecv error: recv got EOF from 127.27.185.66:52425 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 29 20:17:31 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:31 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:31.705953 5204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:31.697495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.67:33483)
0929 20:17:31.698281 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:31.698307 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:31.698470 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:31.698824 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:31.698827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:31.699238 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:31.699426 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:31.699435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:31.700390 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:31.700394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:31.701305 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:31.701315 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:31.702063 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:31.702068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:31.702181 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:31.702882 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:31.702900 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:31.705719 (+ 2819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":525,"threads_started":1}
W20260929 20:17:31.706179 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:31.697666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:33483 (local address 127.27.185.126:39341)
0929 20:17:31.697875 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:31.697879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:31.698413 (+ 534us) server_negotiation.cc:407] Connection header received
0929 20:17:31.698623 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:31.698633 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:31.698695 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:31.698775 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:31.699564 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:31.700260 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:31.701438 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:31.701881 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:31.701995 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:31.705936 (+ 3941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.67:33483: BlockingRecv error: recv got EOF from 127.27.185.67:33483 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 20:17:31 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:31 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:31.807083 5205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:31.796128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.65:52699)
0929 20:17:31.796710 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:31.796729 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:31.796874 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:31.797356 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:31.797361 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:31.797874 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:31.798130 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:31.798144 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:31.799657 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:31.799662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:31.801005 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:31.801020 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:31.802104 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:31.802109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:31.802282 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:31.802949 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:31.802972 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:31.806733 (+ 3761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":287,"threads_started":1}
W20260929 20:17:31.807207 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:31.796211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:52699 (local address 127.27.185.126:39341)
0929 20:17:31.796490 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:31.796496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:31.796870 (+ 374us) server_negotiation.cc:407] Connection header received
0929 20:17:31.797042 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:31.797048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:31.797187 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:31.797305 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:31.798299 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:31.799498 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:31.801160 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:31.801756 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:31.801846 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:31.806908 (+ 5062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.65:52699: BlockingRecv error: recv got EOF from 127.27.185.65:52699 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 20:17:32 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:32 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:32.267280 5206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:32.258836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.66:60657)
0929 20:17:32.259423 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:32.259441 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:32.259557 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:32.259868 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:32.259872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:32.260266 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:32.260441 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:32.260453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:32.261445 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:32.261451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:32.262512 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:32.262525 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:32.263263 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:32.263268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:32.263513 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:32.264134 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:32.264152 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:32.267051 (+ 2899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":310,"threads_started":1}
W20260929 20:17:32.267555 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:32.258876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:60657 (local address 127.27.185.126:39341)
0929 20:17:32.259064 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:32.259069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:32.259567 (+ 498us) server_negotiation.cc:407] Connection header received
0929 20:17:32.259665 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:32.259669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:32.259736 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:32.259824 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:32.260572 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:32.261310 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:32.262647 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:32.263114 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:32.263212 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:32.267249 (+ 4037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.66:60657: BlockingRecv error: recv got EOF from 127.27.185.66:60657 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 20:17:32 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:32 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:32.719760 5207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:32.707063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.67:45023)
0929 20:17:32.707660 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:32.707677 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:32.707834 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:32.708207 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:32.708211 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:32.708763 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:32.708958 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:32.708968 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:32.710098 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:32.710105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:32.711052 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:32.711064 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:32.711920 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:32.711926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:32.712116 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:32.716322 (+ 4206us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:32.716349 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:32.719351 (+ 3002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":373,"threads_started":1}
W20260929 20:17:32.719856 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:32.707181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:45023 (local address 127.27.185.126:39341)
0929 20:17:32.707365 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:32.707370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:32.707766 (+ 396us) server_negotiation.cc:407] Connection header received
0929 20:17:32.707977 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:32.707981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:32.708054 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:32.708171 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:32.709136 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:32.709960 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:32.711206 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:32.711709 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:32.711885 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:32.719624 (+ 7739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.67:45023: BlockingRecv error: recv got EOF from 127.27.185.67:45023 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 20:17:32 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:32 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:32.818251 5208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:32.808255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.65:43125)
0929 20:17:32.809467 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:32.809488 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:32.809626 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:32.810035 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:32.810041 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:32.810590 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:32.810760 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:32.810770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:32.811937 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:32.811942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:32.813065 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:32.813079 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:32.814088 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:32.814093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:32.814222 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:32.814931 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:32.814952 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:32.817788 (+ 2836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1094,"thread_start_us":773,"threads_started":1}
W20260929 20:17:32.818492 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:32.808560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:43125 (local address 127.27.185.126:39341)
0929 20:17:32.808765 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:32.808775 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:32.809582 (+ 807us) server_negotiation.cc:407] Connection header received
0929 20:17:32.809774 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:32.809779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:32.809861 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:32.809991 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:32.810888 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:32.811791 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:32.813469 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:32.813926 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:32.814033 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:32.818080 (+ 4047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.65:43125: BlockingRecv error: recv got EOF from 127.27.185.65:43125 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 20:17:33 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:33 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:33.277211 5209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:33.268268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.66:40791)
0929 20:17:33.268810 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:33.268827 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:33.268944 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:33.269360 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:33.269365 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:33.269758 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:33.269918 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:33.269928 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:33.270957 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:33.270960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:33.271997 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:33.272011 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:33.272793 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:33.272798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:33.273042 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:33.273730 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:33.273749 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:33.276984 (+ 3235us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":313,"threads_started":1}
W20260929 20:17:33.277344 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:33.268397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:40791 (local address 127.27.185.126:39341)
0929 20:17:33.268573 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:33.268577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:33.268905 (+ 328us) server_negotiation.cc:407] Connection header received
0929 20:17:33.269072 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:33.269076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:33.269155 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:33.269253 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:33.270074 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:33.270815 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:33.272182 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:33.272619 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:33.272721 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:33.277145 (+ 4424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.66:40791: BlockingRecv error: recv got EOF from 127.27.185.66:40791 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 20:17:33 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:33 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:33.729694 5210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:33.720858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.67:60929)
0929 20:17:33.721339 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:33.721356 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:33.721506 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:33.721861 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:33.721864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:33.722224 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:33.722367 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:33.722376 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:33.723654 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:33.723659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:33.724627 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:33.724640 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:33.725308 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:33.725312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:33.725464 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:33.726093 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:33.726111 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:33.729382 (+ 3271us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":274,"threads_started":1}
W20260929 20:17:33.729962 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:33.721078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:60929 (local address 127.27.185.126:39341)
0929 20:17:33.721339 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:33.721345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:33.721448 (+ 103us) server_negotiation.cc:407] Connection header received
0929 20:17:33.721654 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:33.721659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:33.721746 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:33.721829 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:33.722686 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:33.723511 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:33.724771 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:33.725167 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:33.725263 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:33.729698 (+ 4435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.67:60929: BlockingRecv error: recv got EOF from 127.27.185.67:60929 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 29 20:17:33 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:33 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:33.828399 5211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:33.819431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.65:44251)
0929 20:17:33.820012 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:33.820029 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:33.820145 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:33.820439 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:33.820442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:33.820878 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:33.821065 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:33.821074 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:33.822231 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:33.822236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:33.823159 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:33.823172 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:33.823996 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:33.824001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:33.824135 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:33.824820 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:33.824839 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:33.828111 (+ 3272us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":320,"threads_started":1}
W20260929 20:17:33.828573 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:33.819469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:44251 (local address 127.27.185.126:39341)
0929 20:17:33.819650 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:33.819655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:33.820104 (+ 449us) server_negotiation.cc:407] Connection header received
0929 20:17:33.820241 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:33.820245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:33.820309 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:33.820393 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:33.821213 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:33.822089 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:33.823302 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:33.823849 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:33.823952 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:33.828329 (+ 4377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.65:44251: BlockingRecv error: recv got EOF from 127.27.185.65:44251 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 20:17:34 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:34 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:34.287057 5212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:34.278349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.66:35809)
0929 20:17:34.278852 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:34.278869 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:34.279012 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:34.279362 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:34.279368 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:34.279828 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:34.280020 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:34.280031 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:34.281093 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:34.281097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:34.282137 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:34.282150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:34.283040 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:34.283046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:34.283251 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:34.283994 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:34.284020 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:34.286814 (+ 2794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":258,"threads_started":1}
W20260929 20:17:34.287245 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:34.278445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:35809 (local address 127.27.185.126:39341)
0929 20:17:34.278695 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:34.278701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:34.278984 (+ 283us) server_negotiation.cc:407] Connection header received
0929 20:17:34.279129 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:34.279133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:34.279197 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:34.279283 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:34.280153 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:34.280965 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:34.282277 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:34.282872 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:34.282996 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:34.286997 (+ 4001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.66:35809: BlockingRecv error: recv got EOF from 127.27.185.66:35809 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 29 20:17:34 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:34 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:34.739024 5213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:34.730703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.67:40607)
0929 20:17:34.731228 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:34.731245 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:34.731364 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:34.731777 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:34.731781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:34.732325 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:34.732522 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:34.732531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:34.733573 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:34.733577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:34.734474 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:34.734486 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:34.735142 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:34.735146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:34.735267 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:34.735897 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:34.735921 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:34.738766 (+ 2845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":295,"threads_started":1}
W20260929 20:17:34.739190 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:34.730815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:40607 (local address 127.27.185.126:39341)
0929 20:17:34.731057 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:34.731062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:34.731329 (+ 267us) server_negotiation.cc:407] Connection header received
0929 20:17:34.731540 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:34.731545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:34.731612 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:34.731697 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:34.732670 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:34.733443 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:34.734615 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:34.735008 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:34.735103 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:34.738980 (+ 3877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.67:40607: BlockingRecv error: recv got EOF from 127.27.185.67:40607 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 29 20:17:34 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:34 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:34.838696 5214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:34.829450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.65:37767)
0929 20:17:34.829961 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:34.829978 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:34.830093 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:34.830441 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:34.830445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:34.830885 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:34.831046 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:34.831056 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:34.832272 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:34.832278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:34.833270 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:34.833286 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:34.834216 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:34.834224 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:34.834429 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:34.835169 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:34.835190 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:34.838354 (+ 3164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":294,"threads_started":1}
W20260929 20:17:34.838860 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:34.829500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:37767 (local address 127.27.185.126:39341)
0929 20:17:34.829766 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:34.829773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:34.830059 (+ 286us) server_negotiation.cc:407] Connection header received
0929 20:17:34.830194 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:34.830197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:34.830261 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:34.830342 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:34.831194 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:34.832038 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:34.833424 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:34.834010 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:34.834147 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:34.838597 (+ 4450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.65:37767: BlockingRecv error: recv got EOF from 127.27.185.65:37767 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 29 20:17:35 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:35 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:35.297048 5215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:35.288091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.66:55153)
0929 20:17:35.288669 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:35.288686 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:35.288856 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:35.289209 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:35.289214 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:35.289651 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:35.289808 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:35.289817 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:35.291019 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:35.291024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:35.292443 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:35.292456 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:35.293147 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:35.293151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:35.293282 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:35.293969 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:35.293988 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:35.296819 (+ 2831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":365,"threads_started":1}
W20260929 20:17:35.297276 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:35.288174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:55153 (local address 127.27.185.126:39341)
0929 20:17:35.288403 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:35.288411 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:35.288779 (+ 368us) server_negotiation.cc:407] Connection header received
0929 20:17:35.289007 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:35.289011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:35.289073 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:35.289175 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:35.289952 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:35.290840 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:35.292589 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:35.293019 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:35.293114 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:35.297024 (+ 3910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.66:55153: BlockingRecv error: recv got EOF from 127.27.185.66:55153 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 20:17:35 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:35 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:35.748682 5216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:35.740137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.67:38921)
0929 20:17:35.740822 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:35.740839 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:35.741000 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:35.741375 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:35.741379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:35.741803 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:35.741955 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:35.741964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:35.742962 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:35.742969 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:35.743932 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:35.743944 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:35.744681 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:35.744686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:35.744881 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:35.745456 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:35.745474 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:35.748390 (+ 2916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":436,"threads_started":1}
W20260929 20:17:35.748997 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:35.740227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:38921 (local address 127.27.185.126:39341)
0929 20:17:35.740456 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:35.740462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:35.740932 (+ 470us) server_negotiation.cc:407] Connection header received
0929 20:17:35.741141 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:35.741151 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:35.741215 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:35.741333 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:35.742129 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:35.742813 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:35.744081 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:35.744525 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:35.744670 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:35.748688 (+ 4018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.67:38921: BlockingRecv error: recv got EOF from 127.27.185.67:38921 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 20:17:35 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:35 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:35.848574 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:35.840058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.65:36333)
0929 20:17:35.840632 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:35.840650 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:35.840770 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:35.841090 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:35.841094 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:35.841526 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:35.841697 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:35.841706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:35.842771 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:35.842776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:35.843851 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:35.843865 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:35.844621 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:35.844626 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:35.844741 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:35.845326 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:35.845344 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:35.848320 (+ 2976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":302,"threads_started":1}
W20260929 20:17:35.848778 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:35.840131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:36333 (local address 127.27.185.126:39341)
0929 20:17:35.840353 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:35.840358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:35.840762 (+ 404us) server_negotiation.cc:407] Connection header received
0929 20:17:35.840878 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:35.840882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:35.840953 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:35.841040 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:35.841843 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:35.842605 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:35.844029 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:35.844465 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:35.844563 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:35.848535 (+ 3972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.65:36333: BlockingRecv error: recv got EOF from 127.27.185.65:36333 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 20:17:36 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:36 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:36.306838 5218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:36.298050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.66:38831)
0929 20:17:36.298656 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:36.298675 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:36.298839 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:36.299225 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:36.299229 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:36.299649 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:36.299812 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:36.299821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:36.301019 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:36.301023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:36.302027 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:36.302040 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:36.302856 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:36.302861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:36.302987 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:36.303808 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:36.303827 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:36.306593 (+ 2766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":305,"threads_started":1}
W20260929 20:17:36.307060 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:36.298167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:38831 (local address 127.27.185.126:39341)
0929 20:17:36.298378 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:36.298385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:36.298791 (+ 406us) server_negotiation.cc:407] Connection header received
0929 20:17:36.298965 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:36.298972 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:36.299072 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:36.299174 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:36.299972 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:36.300843 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:36.302177 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:36.302649 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:36.302752 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:36.306796 (+ 4044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.66:38831: BlockingRecv error: recv got EOF from 127.27.185.66:38831 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 29 20:17:36 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:36 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:36.758562 5219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:36.749678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.67:46221)
0929 20:17:36.750210 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:36.750227 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:36.750360 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:36.750667 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:36.750670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:36.751068 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:36.751226 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:36.751233 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:36.752518 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:36.752522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:36.753483 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:36.753494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:36.754280 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:36.754284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:36.754457 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:36.755029 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:36.755053 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:36.758279 (+ 3226us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":297,"threads_started":1}
W20260929 20:17:36.758716 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:36.749823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:46221 (local address 127.27.185.126:39341)
0929 20:17:36.750081 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:36.750088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:36.750315 (+ 227us) server_negotiation.cc:407] Connection header received
0929 20:17:36.750473 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:36.750477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:36.750539 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:36.750620 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:36.751450 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:36.752381 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:36.753608 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:36.754111 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:36.754209 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:36.758487 (+ 4278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.67:46221: BlockingRecv error: recv got EOF from 127.27.185.67:46221 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
Sep 29 20:17:36 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:36 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:36.858134 5220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:36.849645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.65:55857)
0929 20:17:36.850187 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:36.850202 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:36.850364 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:36.850717 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:36.850722 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:36.851143 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:36.851329 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:36.851338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:36.852397 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:36.852405 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:36.853329 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:36.853340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:36.854048 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:36.854053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:36.854229 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:36.854874 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:36.854892 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:36.857758 (+ 2866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":343,"threads_started":1}
W20260929 20:17:36.858208 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:36.849692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:55857 (local address 127.27.185.126:39341)
0929 20:17:36.850054 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:36.850060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:36.850294 (+ 234us) server_negotiation.cc:407] Connection header received
0929 20:17:36.850498 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:36.850504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:36.850579 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:36.850683 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:36.851498 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:36.852253 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:36.853476 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:36.853892 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:36.854025 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:36.857967 (+ 3942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.65:55857: BlockingRecv error: recv got EOF from 127.27.185.65:55857 (error 108)
Metrics: {"server-negotiator.queue_time_us":216}
Sep 29 20:17:37 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:37 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:37.317096 5221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:37.307885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.66:60891)
0929 20:17:37.308367 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:37.308384 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:37.308576 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:37.308929 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:37.308934 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:37.309381 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:37.309565 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:37.309574 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:37.310928 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:37.310932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:37.312123 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:37.312137 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:37.312956 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:37.312962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:37.313145 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:37.313824 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:37.313843 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:37.316746 (+ 2903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":253,"threads_started":1}
W20260929 20:17:37.317202 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:37.307981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:60891 (local address 127.27.185.126:39341)
0929 20:17:37.308218 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:37.308225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:37.308503 (+ 278us) server_negotiation.cc:407] Connection header received
0929 20:17:37.308713 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:37.308718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:37.308791 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:37.308896 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:37.309716 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:37.310691 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:37.312264 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:37.312807 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:37.312950 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:37.316974 (+ 4024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.66:60891: BlockingRecv error: recv got EOF from 127.27.185.66:60891 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 29 20:17:37 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:37 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.67@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:37.774039 5222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:37.759805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.67:51819)
0929 20:17:37.760279 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:37.760297 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:37.760433 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:37.760851 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:37.760855 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:37.761233 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:37.761462 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:37.761472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:37.762568 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:37.762575 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:37.763510 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:37.763522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:37.764266 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:37.764272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:37.764455 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:37.770489 (+ 6034us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:37.770510 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:37.773665 (+ 3155us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":264,"threads_started":1}
W20260929 20:17:37.774245 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:37.760010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:51819 (local address 127.27.185.126:39341)
0929 20:17:37.760264 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:37.760273 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:37.760382 (+ 109us) server_negotiation.cc:407] Connection header received
0929 20:17:37.760602 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:37.760607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:37.760675 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:37.760772 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:37.761644 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:37.762416 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:37.763661 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:37.764116 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:37.764255 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:37.773945 (+ 9690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.67:51819: BlockingRecv error: recv got EOF from 127.27.185.67:51819 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 29 20:17:37 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:37 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.65@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:37.868851 5223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:37.859641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.65:41613)
0929 20:17:37.860203 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:37.860219 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:37.860379 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:37.860853 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:37.860858 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:37.861418 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:37.861665 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:37.861675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:37.863130 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:37.863133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:37.864210 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:37.864223 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:37.864916 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:37.864920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:37.865035 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:37.865590 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:37.865607 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:37.868596 (+ 2989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":319,"threads_started":1}
W20260929 20:17:37.869091 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:37.859791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:41613 (local address 127.27.185.126:39341)
0929 20:17:37.860045 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:37.860052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:37.860315 (+ 263us) server_negotiation.cc:407] Connection header received
0929 20:17:37.860575 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:37.860580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:37.860673 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:37.860844 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:37.861839 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:37.862980 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:37.864350 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:37.864776 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:37.864892 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:37.868837 (+ 3945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.65:41613: BlockingRecv error: recv got EOF from 127.27.185.65:41613 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
Sep 29 20:17:38 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:17:38 dist-test-slave-4cx3 krb5kdc[4178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.185.66@KRBTEST.COM for oryx/127.27.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:17:38.326831 5224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:38.318247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:39341 (local address 127.27.185.66:39523)
0929 20:17:38.318844 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:38.318860 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:38.318989 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:38.319285 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:38.319288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:38.319773 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:38.319958 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:38.319968 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:38.321004 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:38.321008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:38.321970 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:38.321982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:38.322827 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:38.322831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:38.323031 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:38.323803 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:38.323821 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:38.326591 (+ 2770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.185.126:39341: Server oryx/127.27.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":346,"threads_started":1}
W20260929 20:17:38.327004 4915 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:38.318382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:39523 (local address 127.27.185.126:39341)
0929 20:17:38.318580 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:38.318584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:38.318938 (+ 354us) server_negotiation.cc:407] Connection header received
0929 20:17:38.319094 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:38.319097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:38.319161 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:38.319244 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:38.320093 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:38.320851 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:38.322104 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:38.322675 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:38.322816 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:38.326779 (+ 3963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.185.66:39523: BlockingRecv error: recv got EOF from 127.27.185.66:39523 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:17:38.601745 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 4765
I20260929 20:17:38.614284 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:38.602958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58834 (local address 127.27.185.65:36773)
0929 20:17:38.603527 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:38.603535 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:38.603562 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:17:38.603623 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:38.603630 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:38.603847 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:38.604011 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:38.605445 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:38.606455 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:38.607886 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:38.608429 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:38.608530 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:38.611064 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:38.611071 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:38.611085 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:38.611130 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:38.613798 (+ 2668us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.185.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 20:17:38.614063 (+ 265us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58834: Request ticket server kudu/127.27.185.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":265,"threads_started":1}
W20260929 20:17:38.614367 5226 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58834: Request ticket server kudu/127.27.185.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 20:17:38.614643 28389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 4765: Not authorized: Client connection negotiation failed: client connection to 127.27.185.65:36773: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.185.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 20:17:38.614862 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 4765
I20260929 20:17:38.631655 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 4917
I20260929 20:17:38.644239 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:38.632750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48516 (local address 127.27.185.66:36251)
0929 20:17:38.633313 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:38.633320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:38.633349 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:17:38.633442 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:38.633449 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:38.633679 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:38.633827 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:38.635061 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:38.636240 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:38.637560 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:38.638261 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:38.638360 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:38.640864 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:38.640873 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:38.640892 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:38.640929 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:38.643698 (+ 2769us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.185.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 20:17:38.643969 (+ 271us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48516: Request ticket server kudu/127.27.185.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":264,"threads_started":1}
W20260929 20:17:38.644343 5227 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48516: Request ticket server kudu/127.27.185.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 20:17:38.644394 28389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 4917: Not authorized: Client connection negotiation failed: client connection to 127.27.185.66:36251: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.185.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 20:17:38.644608 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 4917
I20260929 20:17:38.660909 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 5048
I20260929 20:17:38.672989 5228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:38.661943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42032 (local address 127.27.185.67:46507)
0929 20:17:38.662570 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:38.662578 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:38.662605 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:17:38.662672 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:38.662678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:38.662922 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:38.663075 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:38.664480 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:38.665516 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:38.666705 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:38.667286 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:38.667512 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:38.670009 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:38.670018 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:38.670032 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:38.670065 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:38.672489 (+ 2424us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.185.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 20:17:38.672737 (+ 248us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42032: Request ticket server kudu/127.27.185.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":334,"threads_started":1}
W20260929 20:17:38.673126 5228 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42032: Request ticket server kudu/127.27.185.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 20:17:38.673148 28389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 5048: Not authorized: Client connection negotiation failed: client connection to 127.27.185.67:46507: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.185.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 20:17:38.673301 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 5048
I20260929 20:17:38.689685 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 4696
I20260929 20:17:38.691039 4758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:38.806813 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 4696
2026-09-29T20:17:38Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21143 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T20:17:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:17:38Z Disabled control of system clock
I20260929 20:17:38.883333 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:40103
--webserver_interface=127.27.185.126
--webserver_port=0
--builtin_ntp_servers=127.27.185.84:34187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:40103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 20:17:39.220472 5233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:39.220757 5233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:39.220896 5233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:39.228103 5233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:17:39.228199 5233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:39.228291 5233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:39.228410 5233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:17:39.228470 5233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:17:39.237610 5233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:34187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:40103
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:40103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:39.238903 5233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:39.240698 5233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:39.253054 5238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:39.252988 5239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:39.253088 5241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:39.253582 5233 server_base.cc:1034] running on GCE node
I20260929 20:17:39.254366 5233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:39.255873 5233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:39.257105 5233 hybrid_clock.cc:648] HybridClock initialized: now 1790713059257053 us; error 47 us; skew 500 ppm
I20260929 20:17:39.257500 5233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:17:39.259809 5233 webserver.cc:460] Webserver started at http://127.27.185.126:42517/ using document root <none> and password file <none>
I20260929 20:17:39.260531 5233 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:39.260666 5233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:39.260996 5233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:17:39.264003 5233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "481b5b72f167444c8b0af9ad99ee1796"
format_stamp: "Formatted at 2026-09-29 20:17:39 on dist-test-slave-4cx3"
server_key: "15a8a9e312c28b35884d7f1f30a951a7"
server_key_iv: "45f77dbcc18af2293d751edc8054c552"
server_key_version: "encryptionkey@0"
I20260929 20:17:39.264780 5233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "481b5b72f167444c8b0af9ad99ee1796"
format_stamp: "Formatted at 2026-09-29 20:17:39 on dist-test-slave-4cx3"
server_key: "15a8a9e312c28b35884d7f1f30a951a7"
server_key_iv: "45f77dbcc18af2293d751edc8054c552"
server_key_version: "encryptionkey@0"
I20260929 20:17:39.269852 5233 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:17:39.273283 5247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:39.274682 5233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:17:39.274869 5233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "481b5b72f167444c8b0af9ad99ee1796"
format_stamp: "Formatted at 2026-09-29 20:17:39 on dist-test-slave-4cx3"
server_key: "15a8a9e312c28b35884d7f1f30a951a7"
server_key_iv: "45f77dbcc18af2293d751edc8054c552"
server_key_version: "encryptionkey@0"
I20260929 20:17:39.275096 5233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:39.302699 5233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:39.303659 5233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:39.303997 5233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:39.322921 5233 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:40103
I20260929 20:17:39.322957 5298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:40103 every 8 connection(s)
I20260929 20:17:39.324360 5233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:17:39.328187 5299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:39.334695 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 5233
I20260929 20:17:39.334923 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:17:39.335820 28389 external_mini_cluster.cc:1428] Setting key 3f8283c938e8a11fa26755351a837b8d
I20260929 20:17:39.339625 5299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 481b5b72f167444c8b0af9ad99ee1796: Bootstrap starting.
I20260929 20:17:39.343150 5299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 481b5b72f167444c8b0af9ad99ee1796: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:39.344419 5299 log.cc:824] T 00000000000000000000000000000000 P 481b5b72f167444c8b0af9ad99ee1796: Log is configured to *not* fsync() on all Append() calls
I20260929 20:17:39.348040 5299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 481b5b72f167444c8b0af9ad99ee1796: No bootstrap required, opened a new log
I20260929 20:17:39.350492 5302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:39.338029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43944 (local address 127.27.185.126:40103)
0929 20:17:39.338995 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:39.339006 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:39.339048 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:17:39.340774 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:39.340798 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:39.341281 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:39.341609 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:39.342298 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:39.343867 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:39.345669 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:39.346435 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:39.346576 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:39.348018 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:39.348029 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:39.348043 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:17:39.348689 (+ 646us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:39.348770 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:17:39.348792 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:39.348902 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:39.349115 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 20:17:39.349491 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":357,"threads_started":1}
I20260929 20:17:39.356033 5299 raft_consensus.cc:348] T 00000000000000000000000000000000 P 481b5b72f167444c8b0af9ad99ee1796 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481b5b72f167444c8b0af9ad99ee1796" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 40103 } }
I20260929 20:17:39.356412 5299 raft_consensus.cc:374] T 00000000000000000000000000000000 P 481b5b72f167444c8b0af9ad99ee1796 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:39.356520 5299 raft_consensus.cc:729] T 00000000000000000000000000000000 P 481b5b72f167444c8b0af9ad99ee1796 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 481b5b72f167444c8b0af9ad99ee1796, State: Initialized, Role: FOLLOWER
I20260929 20:17:39.357141 5299 consensus_queue.cc:260] T 00000000000000000000000000000000 P 481b5b72f167444c8b0af9ad99ee1796 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481b5b72f167444c8b0af9ad99ee1796" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 40103 } }
I20260929 20:17:39.357419 5299 raft_consensus.cc:388] T 00000000000000000000000000000000 P 481b5b72f167444c8b0af9ad99ee1796 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:17:39.357561 5299 raft_consensus.cc:482] T 00000000000000000000000000000000 P 481b5b72f167444c8b0af9ad99ee1796 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:17:39.357738 5299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 481b5b72f167444c8b0af9ad99ee1796 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:17:39.360031 5299 raft_consensus.cc:504] T 00000000000000000000000000000000 P 481b5b72f167444c8b0af9ad99ee1796 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481b5b72f167444c8b0af9ad99ee1796" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 40103 } }
I20260929 20:17:39.360538 5299 leader_election.cc:302] T 00000000000000000000000000000000 P 481b5b72f167444c8b0af9ad99ee1796 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 481b5b72f167444c8b0af9ad99ee1796; no voters:
I20260929 20:17:39.361131 5299 leader_election.cc:288] T 00000000000000000000000000000000 P 481b5b72f167444c8b0af9ad99ee1796 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:17:39.361291 5304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 481b5b72f167444c8b0af9ad99ee1796 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:17:39.362048 5304 raft_consensus.cc:686] T 00000000000000000000000000000000 P 481b5b72f167444c8b0af9ad99ee1796 [term 1 LEADER]: Becoming Leader. State: Replica: 481b5b72f167444c8b0af9ad99ee1796, State: Running, Role: LEADER
I20260929 20:17:39.362713 5304 consensus_queue.cc:237] T 00000000000000000000000000000000 P 481b5b72f167444c8b0af9ad99ee1796 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481b5b72f167444c8b0af9ad99ee1796" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 40103 } }
I20260929 20:17:39.363334 5299 sys_catalog.cc:564] T 00000000000000000000000000000000 P 481b5b72f167444c8b0af9ad99ee1796 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:17:39.365465 5306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 481b5b72f167444c8b0af9ad99ee1796 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 481b5b72f167444c8b0af9ad99ee1796. Latest consensus state: current_term: 1 leader_uuid: "481b5b72f167444c8b0af9ad99ee1796" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481b5b72f167444c8b0af9ad99ee1796" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 40103 } } }
I20260929 20:17:39.365975 5306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 481b5b72f167444c8b0af9ad99ee1796 [sys.catalog]: This master's current role is: LEADER
I20260929 20:17:39.366328 5305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 481b5b72f167444c8b0af9ad99ee1796 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "481b5b72f167444c8b0af9ad99ee1796" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481b5b72f167444c8b0af9ad99ee1796" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 40103 } } }
I20260929 20:17:39.367337 5305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 481b5b72f167444c8b0af9ad99ee1796 [sys.catalog]: This master's current role is: LEADER
I20260929 20:17:39.369118 5309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:17:39.376113 5309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:17:39.386606 5309 catalog_manager.cc:1347] Generated new cluster ID: c28600090e334e68af00f80ecd69f817
I20260929 20:17:39.386736 5309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:17:39.406131 5309 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:17:39.407756 5309 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:17:39.418996 5309 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 481b5b72f167444c8b0af9ad99ee1796: Generated new TSK 0
I20260929 20:17:39.419906 5309 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:17:39.426318 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:0
--local_ip_for_outbound_sockets=127.27.185.65
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:40103
--builtin_ntp_servers=127.27.185.84:34187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 20:17:39.765333 5323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:39.765672 5323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:39.765827 5323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:39.773386 5323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:39.773527 5323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:39.773700 5323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:17:39.784056 5323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:34187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:40103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:39.785420 5323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:39.787235 5323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:39.800684 5328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:39.800714 5331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:39.800819 5329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:39.801646 5323 server_base.cc:1034] running on GCE node
I20260929 20:17:39.802237 5323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:39.803181 5323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:39.804422 5323 hybrid_clock.cc:648] HybridClock initialized: now 1790713059804386 us; error 66 us; skew 500 ppm
I20260929 20:17:39.804761 5323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:17:39.807073 5323 webserver.cc:460] Webserver started at http://127.27.185.65:41979/ using document root <none> and password file <none>
I20260929 20:17:39.807852 5323 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:39.807965 5323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:39.808341 5323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:17:39.811138 5323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/data/instance:
uuid: "4895f045ea3044e9bda265a0a9359559"
format_stamp: "Formatted at 2026-09-29 20:17:39 on dist-test-slave-4cx3"
server_key: "7ae97f73024fb908bea941f0d5ef3aee"
server_key_iv: "da901bf324ca7e124ea8c35c04fe51ea"
server_key_version: "encryptionkey@0"
I20260929 20:17:39.811991 5323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance:
uuid: "4895f045ea3044e9bda265a0a9359559"
format_stamp: "Formatted at 2026-09-29 20:17:39 on dist-test-slave-4cx3"
server_key: "7ae97f73024fb908bea941f0d5ef3aee"
server_key_iv: "da901bf324ca7e124ea8c35c04fe51ea"
server_key_version: "encryptionkey@0"
I20260929 20:17:39.817075 5323 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:17:39.820536 5337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:39.822006 5323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:17:39.822213 5323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "4895f045ea3044e9bda265a0a9359559"
format_stamp: "Formatted at 2026-09-29 20:17:39 on dist-test-slave-4cx3"
server_key: "7ae97f73024fb908bea941f0d5ef3aee"
server_key_iv: "da901bf324ca7e124ea8c35c04fe51ea"
server_key_version: "encryptionkey@0"
I20260929 20:17:39.822481 5323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:39.853425 5323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:39.854368 5323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:39.854679 5323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:39.855726 5323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:17:39.857184 5323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:17:39.857307 5323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:39.857455 5323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:17:39.857542 5323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:39.892617 5323 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:36413
I20260929 20:17:39.892715 5449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:36413 every 8 connection(s)
I20260929 20:17:39.893926 5323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:17:39.903132 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 5323
I20260929 20:17:39.903374 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance
I20260929 20:17:39.904064 28389 external_mini_cluster.cc:1428] Setting key 50c355592865932294836bdaffc510c4
I20260929 20:17:39.906879 5452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:39.896627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:46607 (local address 127.27.185.126:40103)
0929 20:17:39.897111 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:39.897116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:39.898197 (+ 1081us) server_negotiation.cc:407] Connection header received
0929 20:17:39.900096 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:39.900101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:39.900198 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:39.900315 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:39.901437 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:39.902199 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:39.903940 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:39.904407 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:39.904551 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:39.906038 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:39.906043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:39.906046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:17:39.906063 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:39.906105 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:17:39.906118 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:39.906212 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:39.906439 (+ 227us) server_negotiation.cc:299] Negotiation successful
0929 20:17:39.906539 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":271,"threads_started":1}
I20260929 20:17:39.907119 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:0
--local_ip_for_outbound_sockets=127.27.185.66
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:40103
--builtin_ntp_servers=127.27.185.84:34187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 20:17:39.908118 5453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:39.897127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:40103 (local address 127.27.185.65:46607)
0929 20:17:39.898033 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:39.898071 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:39.899811 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:39.900582 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:39.900591 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:39.900685 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:39.901268 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:39.901285 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:39.902350 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:39.902360 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:39.903792 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:39.903804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:39.904643 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:39.904653 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:39.904901 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:39.905665 (+ 764us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:17:39.905669 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:39.905723 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:17:39.905835 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:39.906212 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:39.906225 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:39.906236 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 20:17:39.906432 (+ 196us) client_negotiation.cc:240] Negotiation successful
0929 20:17:39.906864 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":281,"threads_started":1}
I20260929 20:17:39.909737 5450 heartbeater.cc:347] Connected to a master server at 127.27.185.126:40103
I20260929 20:17:39.910094 5450 heartbeater.cc:464] Registering TS with master...
I20260929 20:17:39.911054 5450 heartbeater.cc:511] Master 127.27.185.126:40103 requested a full tablet report, sending...
I20260929 20:17:39.913460 5264 ts_manager.cc:194] Registered new tserver with Master: 4895f045ea3044e9bda265a0a9359559 (127.27.185.65:36413)
I20260929 20:17:39.915230 5264 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.185.65:46607
W20260929 20:17:40.240617 5454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:40.240890 5454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:40.241039 5454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:40.248358 5454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:40.248461 5454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:40.248624 5454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:17:40.258059 5454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:34187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:40103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:40.259447 5454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:40.261358 5454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:40.274766 5462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:40.274861 5459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:40.274837 5460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:40.275622 5454 server_base.cc:1034] running on GCE node
I20260929 20:17:40.276440 5454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:40.277628 5454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:40.278841 5454 hybrid_clock.cc:648] HybridClock initialized: now 1790713060278784 us; error 47 us; skew 500 ppm
I20260929 20:17:40.279168 5454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:17:40.281490 5454 webserver.cc:460] Webserver started at http://127.27.185.66:38289/ using document root <none> and password file <none>
I20260929 20:17:40.282197 5454 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:40.282330 5454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:40.282661 5454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:17:40.285490 5454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/data/instance:
uuid: "b18e2760e2854b5a8aff3e3de7b0b01d"
format_stamp: "Formatted at 2026-09-29 20:17:40 on dist-test-slave-4cx3"
server_key: "53c6e3f870528473d4f665e317ba30ef"
server_key_iv: "2b8f95187dc52fcd64313cf2356eca71"
server_key_version: "encryptionkey@0"
I20260929 20:17:40.286291 5454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance:
uuid: "b18e2760e2854b5a8aff3e3de7b0b01d"
format_stamp: "Formatted at 2026-09-29 20:17:40 on dist-test-slave-4cx3"
server_key: "53c6e3f870528473d4f665e317ba30ef"
server_key_iv: "2b8f95187dc52fcd64313cf2356eca71"
server_key_version: "encryptionkey@0"
I20260929 20:17:40.291368 5454 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 20:17:40.294816 5468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:40.296245 5454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 20:17:40.296425 5454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "b18e2760e2854b5a8aff3e3de7b0b01d"
format_stamp: "Formatted at 2026-09-29 20:17:40 on dist-test-slave-4cx3"
server_key: "53c6e3f870528473d4f665e317ba30ef"
server_key_iv: "2b8f95187dc52fcd64313cf2356eca71"
server_key_version: "encryptionkey@0"
I20260929 20:17:40.296653 5454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:40.314044 5454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:40.314891 5454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:40.315197 5454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:40.316200 5454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:17:40.317669 5454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:17:40.317790 5454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:40.317926 5454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:17:40.318017 5454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:40.352205 5454 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:44251
I20260929 20:17:40.352238 5580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:44251 every 8 connection(s)
I20260929 20:17:40.353521 5454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:17:40.362290 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 5454
I20260929 20:17:40.362555 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance
I20260929 20:17:40.363071 28389 external_mini_cluster.cc:1428] Setting key 79ecc9d25a78ae59fedc4fc93d901ac5
I20260929 20:17:40.366607 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:0
--local_ip_for_outbound_sockets=127.27.185.67
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:40103
--builtin_ntp_servers=127.27.185.84:34187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 20:17:40.366968 5452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:40.356343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:43199 (local address 127.27.185.126:40103)
0929 20:17:40.356523 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:40.356528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:40.358114 (+ 1586us) server_negotiation.cc:407] Connection header received
0929 20:17:40.359959 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:40.359965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:40.360063 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:40.360176 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:40.361272 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:40.362007 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:40.363829 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:40.364281 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:40.364432 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:40.366081 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:40.366087 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:40.366090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:17:40.366108 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:40.366140 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:17:40.366152 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:40.366222 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:40.366520 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 20:17:40.366604 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 20:17:40.368587 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:40.357040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:40103 (local address 127.27.185.66:43199)
0929 20:17:40.357938 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:40.357981 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:40.359688 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:40.360422 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:40.360430 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:40.360521 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:40.361106 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:40.361124 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:40.362161 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:40.362167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:40.363683 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:40.363696 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:40.364543 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:40.364551 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:40.364882 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:40.365693 (+ 811us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:17:40.365696 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:40.365755 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:17:40.365870 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:40.366257 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:40.366271 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:40.366283 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 20:17:40.366501 (+ 218us) client_negotiation.cc:240] Negotiation successful
0929 20:17:40.367004 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":281,"threads_started":1}
I20260929 20:17:40.370085 5581 heartbeater.cc:347] Connected to a master server at 127.27.185.126:40103
I20260929 20:17:40.370420 5581 heartbeater.cc:464] Registering TS with master...
I20260929 20:17:40.371443 5581 heartbeater.cc:511] Master 127.27.185.126:40103 requested a full tablet report, sending...
I20260929 20:17:40.373059 5264 ts_manager.cc:194] Registered new tserver with Master: b18e2760e2854b5a8aff3e3de7b0b01d (127.27.185.66:44251)
I20260929 20:17:40.373899 5264 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.185.66:43199
W20260929 20:17:40.697669 5584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:40.698017 5584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:40.698180 5584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:40.705551 5584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:40.705700 5584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:40.705879 5584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:17:40.714932 5584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:34187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:40103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:40.716351 5584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:40.718174 5584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:40.731743 5589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:40.731824 5592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:40.731771 5590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:40.732939 5584 server_base.cc:1034] running on GCE node
I20260929 20:17:40.733701 5584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:40.734731 5584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:40.735963 5584 hybrid_clock.cc:648] HybridClock initialized: now 1790713060735913 us; error 46 us; skew 500 ppm
I20260929 20:17:40.736303 5584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:17:40.738770 5584 webserver.cc:460] Webserver started at http://127.27.185.67:33137/ using document root <none> and password file <none>
I20260929 20:17:40.739562 5584 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:40.739696 5584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:40.740033 5584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:17:40.742869 5584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/data/instance:
uuid: "b7b2db43c8074e5ba57006f33f497956"
format_stamp: "Formatted at 2026-09-29 20:17:40 on dist-test-slave-4cx3"
server_key: "8c996f80aa99957b1183f4551344a3ca"
server_key_iv: "9b68f99cd80c021b397bd7111d6469fd"
server_key_version: "encryptionkey@0"
I20260929 20:17:40.743713 5584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance:
uuid: "b7b2db43c8074e5ba57006f33f497956"
format_stamp: "Formatted at 2026-09-29 20:17:40 on dist-test-slave-4cx3"
server_key: "8c996f80aa99957b1183f4551344a3ca"
server_key_iv: "9b68f99cd80c021b397bd7111d6469fd"
server_key_version: "encryptionkey@0"
I20260929 20:17:40.748988 5584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:17:40.752375 5598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:40.753791 5584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:17:40.753986 5584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "b7b2db43c8074e5ba57006f33f497956"
format_stamp: "Formatted at 2026-09-29 20:17:40 on dist-test-slave-4cx3"
server_key: "8c996f80aa99957b1183f4551344a3ca"
server_key_iv: "9b68f99cd80c021b397bd7111d6469fd"
server_key_version: "encryptionkey@0"
I20260929 20:17:40.754223 5584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:40.776391 5584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:40.777374 5584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:40.777711 5584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:40.778833 5584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:17:40.780402 5584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:17:40.780524 5584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:40.780659 5584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:17:40.780747 5584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:40.815080 5584 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:46865
I20260929 20:17:40.815172 5710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:46865 every 8 connection(s)
I20260929 20:17:40.816483 5584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:17:40.821336 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 5584
I20260929 20:17:40.821547 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance
I20260929 20:17:40.822095 28389 external_mini_cluster.cc:1428] Setting key a6b345aa80b3bf513ba9de7f396e89e0
I20260929 20:17:40.830263 5452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:40.819080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:51979 (local address 127.27.185.126:40103)
0929 20:17:40.819258 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:40.819263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:40.820805 (+ 1542us) server_negotiation.cc:407] Connection header received
0929 20:17:40.823262 (+ 2457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:40.823267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:40.823337 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:40.823504 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:40.824933 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:40.825903 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:40.827702 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:40.828121 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:40.828222 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:40.829581 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:40.829586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:40.829589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:17:40.829608 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:40.829638 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:17:40.829650 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:40.829722 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:40.829924 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 20:17:40.830005 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:17:40.831256 5713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:40.819714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:40103 (local address 127.27.185.67:51979)
0929 20:17:40.820659 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:40.820697 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:40.822971 (+ 2274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:40.823776 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:40.823785 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:40.823886 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:40.824730 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:40.824758 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:40.826079 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:40.826083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:40.827572 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:40.827584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:40.828308 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:40.828313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:40.828543 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:40.829232 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:17:40.829234 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:40.829275 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:17:40.829374 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:40.829734 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:40.829742 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:40.829748 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 20:17:40.829891 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 20:17:40.830197 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":274,"threads_started":1}
I20260929 20:17:40.832664 5711 heartbeater.cc:347] Connected to a master server at 127.27.185.126:40103
I20260929 20:17:40.832973 5711 heartbeater.cc:464] Registering TS with master...
I20260929 20:17:40.833768 5711 heartbeater.cc:511] Master 127.27.185.126:40103 requested a full tablet report, sending...
I20260929 20:17:40.835284 5264 ts_manager.cc:194] Registered new tserver with Master: b7b2db43c8074e5ba57006f33f497956 (127.27.185.67:46865)
I20260929 20:17:40.836257 5264 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.185.67:51979
I20260929 20:17:40.839083 28389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 20:17:40.845666 5452 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:40.843791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43948 (local address 127.27.185.126:40103)
0929 20:17:40.843965 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:40.843969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:40.844517 (+ 548us) server_negotiation.cc:407] Connection header received
0929 20:17:40.844719 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:40.844725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:40.844812 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:40.844927 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:40.845381 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43948: BlockingRecv error: recv got EOF from 127.0.0.1:43948 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 20:17:40.846158 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 5323
I20260929 20:17:40.854182 5720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:40.847188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37872 (local address 127.27.185.65:36413)
0929 20:17:40.847771 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:40.847778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:40.847804 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:17:40.847865 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:40.847870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:40.848074 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:40.848203 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:40.848592 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:40.849588 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:40.851041 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:40.851692 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:40.851791 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:40.852817 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:40.852824 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:40.852837 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:17:40.853413 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:40.853458 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:17:40.853487 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:40.853605 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:40.853762 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 20:17:40.853875 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":331,"threads_started":1}
I20260929 20:17:40.855556 5445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:40.937635 5450 heartbeater.cc:503] Master 127.27.185.126:40103 was elected leader, sending a full tablet report...
I20260929 20:17:40.938772 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 5323
W20260929 20:17:40.955299 5249 connection.cc:535] server connection from 127.27.185.65:46607 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 20:17:40.955660 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 5454
I20260929 20:17:40.963999 5722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:40.956638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47490 (local address 127.27.185.66:44251)
0929 20:17:40.957158 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:40.957165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:40.957194 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:17:40.957263 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:40.957270 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:40.957525 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:40.957669 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:40.958084 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:40.959175 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:40.960540 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:40.961100 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:40.961199 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:40.962301 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:40.962310 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:40.962367 (+ 57us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:17:40.962988 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:40.963052 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:17:40.963073 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:40.963167 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:40.963519 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 20:17:40.963692 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":253,"threads_started":1}
I20260929 20:17:40.965556 5576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:41.048300 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 5454
I20260929 20:17:41.064956 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 5584
I20260929 20:17:41.073995 5724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:41.066178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58842 (local address 127.27.185.67:46865)
0929 20:17:41.066897 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:41.066905 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:41.066950 (+ 45us) server_negotiation.cc:407] Connection header received
0929 20:17:41.067011 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:41.067018 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:41.067255 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:41.067453 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:41.068041 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:41.069117 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:41.070692 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:41.071277 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:41.071379 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:41.072538 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:41.072546 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:41.072560 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:17:41.073187 (+ 627us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:41.073235 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:17:41.073255 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:41.073346 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:41.073553 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 20:17:41.073667 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":409,"threads_started":1}
I20260929 20:17:41.075373 5706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:41.157312 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 5584
I20260929 20:17:41.173179 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 5233
I20260929 20:17:41.174456 5294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:41.277748 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 5233
2026-09-29T20:17:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2465 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T20:17:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:17:41Z Disabled control of system clock
I20260929 20:17:41.349408 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:45975
--webserver_interface=127.27.185.126
--webserver_port=0
--builtin_ntp_servers=127.27.185.84:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:45975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 20:17:41.682337 5730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:41.682631 5730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:41.682775 5730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:41.690112 5730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:17:41.690212 5730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:41.690275 5730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:41.690378 5730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:17:41.690466 5730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:17:41.699716 5730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:45975
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:45975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:41.701138 5730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:41.702986 5730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:41.715013 5735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:41.715025 5736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:41.715507 5738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:41.715842 5730 server_base.cc:1034] running on GCE node
I20260929 20:17:41.716645 5730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:41.718065 5730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:41.719301 5730 hybrid_clock.cc:648] HybridClock initialized: now 1790713061719248 us; error 44 us; skew 500 ppm
I20260929 20:17:41.719724 5730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:17:41.722038 5730 webserver.cc:460] Webserver started at http://127.27.185.126:34155/ using document root <none> and password file <none>
I20260929 20:17:41.722770 5730 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:41.722913 5730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:41.723254 5730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:17:41.726347 5730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "15054853ede74867be5c5855ba997853"
format_stamp: "Formatted at 2026-09-29 20:17:41 on dist-test-slave-4cx3"
server_key: "8232b2e1cc1b71f6ba2611a8d6464ef8"
server_key_iv: "b4e23f5d26820757afc8fd08c229e3ef"
server_key_version: "encryptionkey@0"
I20260929 20:17:41.727196 5730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "15054853ede74867be5c5855ba997853"
format_stamp: "Formatted at 2026-09-29 20:17:41 on dist-test-slave-4cx3"
server_key: "8232b2e1cc1b71f6ba2611a8d6464ef8"
server_key_iv: "b4e23f5d26820757afc8fd08c229e3ef"
server_key_version: "encryptionkey@0"
I20260929 20:17:41.732645 5730 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 20:17:41.736284 5745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:41.737736 5730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:17:41.737926 5730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "15054853ede74867be5c5855ba997853"
format_stamp: "Formatted at 2026-09-29 20:17:41 on dist-test-slave-4cx3"
server_key: "8232b2e1cc1b71f6ba2611a8d6464ef8"
server_key_iv: "b4e23f5d26820757afc8fd08c229e3ef"
server_key_version: "encryptionkey@0"
I20260929 20:17:41.738152 5730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:41.744131 5730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:41.744915 5730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:41.745217 5730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:41.764022 5730 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:45975
I20260929 20:17:41.764045 5796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:45975 every 8 connection(s)
I20260929 20:17:41.765673 5730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:17:41.769616 5797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:41.771771 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 5730
I20260929 20:17:41.771987 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:17:41.772799 28389 external_mini_cluster.cc:1428] Setting key a81898cbe6315bdc900c3b82fc6c64d2
I20260929 20:17:41.782078 5800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:41.775475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51436 (local address 127.27.185.126:45975)
0929 20:17:41.776708 (+ 1233us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:41.776720 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:41.776763 (+ 43us) server_negotiation.cc:407] Connection header received
0929 20:17:41.778431 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:41.778455 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:41.778923 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:41.779278 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:41.780433 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:41.780442 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:41.780459 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:17:41.780473 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:41.780535 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:17:41.780551 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:41.780733 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:41.780863 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 20:17:41.781175 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":421,"threads_started":1}
I20260929 20:17:41.782650 5797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15054853ede74867be5c5855ba997853: Bootstrap starting.
I20260929 20:17:41.785688 5797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15054853ede74867be5c5855ba997853: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:41.786680 5797 log.cc:824] T 00000000000000000000000000000000 P 15054853ede74867be5c5855ba997853: Log is configured to *not* fsync() on all Append() calls
I20260929 20:17:41.789433 5797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15054853ede74867be5c5855ba997853: No bootstrap required, opened a new log
I20260929 20:17:41.796689 5797 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15054853ede74867be5c5855ba997853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15054853ede74867be5c5855ba997853" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 45975 } }
I20260929 20:17:41.797094 5797 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15054853ede74867be5c5855ba997853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:41.797221 5797 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15054853ede74867be5c5855ba997853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15054853ede74867be5c5855ba997853, State: Initialized, Role: FOLLOWER
I20260929 20:17:41.797901 5797 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15054853ede74867be5c5855ba997853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15054853ede74867be5c5855ba997853" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 45975 } }
I20260929 20:17:41.798131 5797 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15054853ede74867be5c5855ba997853 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:17:41.798257 5797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15054853ede74867be5c5855ba997853 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:17:41.798432 5797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15054853ede74867be5c5855ba997853 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:17:41.800649 5797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15054853ede74867be5c5855ba997853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15054853ede74867be5c5855ba997853" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 45975 } }
I20260929 20:17:41.801230 5797 leader_election.cc:302] T 00000000000000000000000000000000 P 15054853ede74867be5c5855ba997853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15054853ede74867be5c5855ba997853; no voters:
I20260929 20:17:41.801874 5797 leader_election.cc:288] T 00000000000000000000000000000000 P 15054853ede74867be5c5855ba997853 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:17:41.802053 5802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15054853ede74867be5c5855ba997853 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:17:41.802778 5802 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15054853ede74867be5c5855ba997853 [term 1 LEADER]: Becoming Leader. State: Replica: 15054853ede74867be5c5855ba997853, State: Running, Role: LEADER
I20260929 20:17:41.803311 5802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15054853ede74867be5c5855ba997853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15054853ede74867be5c5855ba997853" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 45975 } }
I20260929 20:17:41.804061 5797 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15054853ede74867be5c5855ba997853 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:17:41.806061 5803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15054853ede74867be5c5855ba997853 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15054853ede74867be5c5855ba997853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15054853ede74867be5c5855ba997853" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 45975 } } }
I20260929 20:17:41.806491 5803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15054853ede74867be5c5855ba997853 [sys.catalog]: This master's current role is: LEADER
I20260929 20:17:41.806396 5804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15054853ede74867be5c5855ba997853 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15054853ede74867be5c5855ba997853. Latest consensus state: current_term: 1 leader_uuid: "15054853ede74867be5c5855ba997853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15054853ede74867be5c5855ba997853" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 45975 } } }
I20260929 20:17:41.806700 5804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15054853ede74867be5c5855ba997853 [sys.catalog]: This master's current role is: LEADER
I20260929 20:17:41.808169 5810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:17:41.813684 5810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:17:41.822474 5810 catalog_manager.cc:1347] Generated new cluster ID: 5f7a2678b54d473bb4697f57ade844d5
I20260929 20:17:41.822585 5810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:17:41.860582 5810 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:17:41.861079 5810 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:17:41.876394 5810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15054853ede74867be5c5855ba997853: Generated new TSK 0
I20260929 20:17:41.877506 5810 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:17:41.888814 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:0
--local_ip_for_outbound_sockets=127.27.185.65
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:45975
--builtin_ntp_servers=127.27.185.84:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 20:17:42.222460 5821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:42.222750 5821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:42.222908 5821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:42.230137 5821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:42.230234 5821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:42.230448 5821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:17:42.239607 5821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:45975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:42.240913 5821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:42.242899 5821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:42.256861 5826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:42.256800 5827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:42.256770 5829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:42.257714 5821 server_base.cc:1034] running on GCE node
I20260929 20:17:42.258378 5821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:42.259552 5821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:42.260967 5821 hybrid_clock.cc:648] HybridClock initialized: now 1790713062260754 us; error 58 us; skew 500 ppm
I20260929 20:17:42.261375 5821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:17:42.263898 5821 webserver.cc:460] Webserver started at http://127.27.185.65:44661/ using document root <none> and password file <none>
I20260929 20:17:42.264621 5821 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:42.264758 5821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:42.265101 5821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:17:42.268126 5821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/data/instance:
uuid: "62d7a0404ab94f09ab7e3bd00606ca94"
format_stamp: "Formatted at 2026-09-29 20:17:42 on dist-test-slave-4cx3"
server_key: "f701a87d44065df215cd3545840f7256"
server_key_iv: "b75cd4ddafbe15168d4bd5f2d0fd3986"
server_key_version: "encryptionkey@0"
I20260929 20:17:42.268980 5821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance:
uuid: "62d7a0404ab94f09ab7e3bd00606ca94"
format_stamp: "Formatted at 2026-09-29 20:17:42 on dist-test-slave-4cx3"
server_key: "f701a87d44065df215cd3545840f7256"
server_key_iv: "b75cd4ddafbe15168d4bd5f2d0fd3986"
server_key_version: "encryptionkey@0"
I20260929 20:17:42.274484 5821 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 20:17:42.278069 5835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:42.279637 5821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:17:42.279819 5821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "62d7a0404ab94f09ab7e3bd00606ca94"
format_stamp: "Formatted at 2026-09-29 20:17:42 on dist-test-slave-4cx3"
server_key: "f701a87d44065df215cd3545840f7256"
server_key_iv: "b75cd4ddafbe15168d4bd5f2d0fd3986"
server_key_version: "encryptionkey@0"
I20260929 20:17:42.280064 5821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:42.285742 5821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:42.286538 5821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:42.286811 5821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:42.287815 5821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:17:42.289242 5821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:17:42.289368 5821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:42.289513 5821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:17:42.289605 5821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:42.324157 5821 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:35811
I20260929 20:17:42.324214 5947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:35811 every 8 connection(s)
I20260929 20:17:42.325527 5821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:17:42.331036 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 5821
I20260929 20:17:42.331238 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance
I20260929 20:17:42.331938 28389 external_mini_cluster.cc:1428] Setting key dd2b82576e2c77d83fe71f6fae25587c
I20260929 20:17:42.334344 5950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:42.328232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:37301 (local address 127.27.185.126:45975)
0929 20:17:42.328686 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:42.328691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:42.329967 (+ 1276us) server_negotiation.cc:407] Connection header received
0929 20:17:42.331878 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:42.331884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:42.331979 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:42.332112 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:42.333566 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:42.333572 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:42.333577 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:17:42.333596 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:42.333647 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:17:42.333659 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:42.333791 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:42.333984 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 20:17:42.334051 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":252,"threads_started":1}
I20260929 20:17:42.334731 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:0
--local_ip_for_outbound_sockets=127.27.185.66
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:45975
--builtin_ntp_servers=127.27.185.84:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 20:17:42.335470 5951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:42.328693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:45975 (local address 127.27.185.65:37301)
0929 20:17:42.329781 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:42.329820 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:42.331594 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:42.332402 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:42.332411 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:42.332517 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:42.333246 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:17:42.333248 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:42.333286 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:17:42.333374 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:42.333758 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:42.333771 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:42.333779 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 20:17:42.334077 (+ 298us) client_negotiation.cc:240] Negotiation successful
0929 20:17:42.334346 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":327,"threads_started":1}
I20260929 20:17:42.337175 5948 heartbeater.cc:347] Connected to a master server at 127.27.185.126:45975
I20260929 20:17:42.337527 5948 heartbeater.cc:464] Registering TS with master...
I20260929 20:17:42.338091 5948 heartbeater.cc:511] Master 127.27.185.126:45975 requested a full tablet report, sending...
I20260929 20:17:42.340399 5762 ts_manager.cc:194] Registered new tserver with Master: 62d7a0404ab94f09ab7e3bd00606ca94 (127.27.185.65:35811)
W20260929 20:17:42.685190 5952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:42.685549 5952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:42.685647 5952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:42.692881 5952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:42.692982 5952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:42.693091 5952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:17:42.701987 5952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:45975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:42.703299 5952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:42.705408 5952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:42.718401 5960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:42.718449 5958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:42.718967 5957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:42.719239 5952 server_base.cc:1034] running on GCE node
I20260929 20:17:42.719928 5952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:42.720961 5952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:42.722195 5952 hybrid_clock.cc:648] HybridClock initialized: now 1790713062722124 us; error 64 us; skew 500 ppm
I20260929 20:17:42.722544 5952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:17:42.725582 5952 webserver.cc:460] Webserver started at http://127.27.185.66:39615/ using document root <none> and password file <none>
I20260929 20:17:42.726400 5952 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:42.726572 5952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:42.726931 5952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:17:42.730146 5952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/data/instance:
uuid: "34ef238932ad404ca6056e1394de3609"
format_stamp: "Formatted at 2026-09-29 20:17:42 on dist-test-slave-4cx3"
server_key: "bd6d17201a45f917616baf6ec5cbd0e3"
server_key_iv: "c3ea706d7a29c6cf69a6465bca6bb5e2"
server_key_version: "encryptionkey@0"
I20260929 20:17:42.731046 5952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance:
uuid: "34ef238932ad404ca6056e1394de3609"
format_stamp: "Formatted at 2026-09-29 20:17:42 on dist-test-slave-4cx3"
server_key: "bd6d17201a45f917616baf6ec5cbd0e3"
server_key_iv: "c3ea706d7a29c6cf69a6465bca6bb5e2"
server_key_version: "encryptionkey@0"
I20260929 20:17:42.736855 5952 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:17:42.740667 5966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:42.742378 5952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:17:42.742601 5952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "34ef238932ad404ca6056e1394de3609"
format_stamp: "Formatted at 2026-09-29 20:17:42 on dist-test-slave-4cx3"
server_key: "bd6d17201a45f917616baf6ec5cbd0e3"
server_key_iv: "c3ea706d7a29c6cf69a6465bca6bb5e2"
server_key_version: "encryptionkey@0"
I20260929 20:17:42.742883 5952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:42.748735 5952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:42.749733 5952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:42.750020 5952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:42.751009 5952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:17:42.752626 5952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:17:42.752748 5952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:42.752884 5952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:17:42.752974 5952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:42.790892 5952 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:35807
I20260929 20:17:42.790953 6078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:35807 every 8 connection(s)
I20260929 20:17:42.792481 5952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:17:42.793485 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 5952
I20260929 20:17:42.793644 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance
I20260929 20:17:42.794227 28389 external_mini_cluster.cc:1428] Setting key 97473d0a306fd33d4b418544efe1fac9
I20260929 20:17:42.797102 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:0
--local_ip_for_outbound_sockets=127.27.185.67
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:45975
--builtin_ntp_servers=127.27.185.84:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 20:17:42.806852 5950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:42.795350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:59769 (local address 127.27.185.126:45975)
0929 20:17:42.795597 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:42.795604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:42.797235 (+ 1631us) server_negotiation.cc:407] Connection header received
0929 20:17:42.799024 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:42.799032 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:42.799103 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:42.799215 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:42.806089 (+ 6874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:42.806094 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:42.806097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:17:42.806118 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:42.806157 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:17:42.806169 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:42.806282 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:42.806564 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 20:17:42.806624 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260929 20:17:42.808370 6081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:42.795965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:45975 (local address 127.27.185.66:59769)
0929 20:17:42.797080 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:42.797132 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:42.798742 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:42.799503 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:42.799511 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:42.799606 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:42.805627 (+ 6021us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:17:42.805634 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:42.805696 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:17:42.805838 (+ 142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:42.806294 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:42.806309 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:42.806322 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 20:17:42.806653 (+ 331us) client_negotiation.cc:240] Negotiation successful
0929 20:17:42.806962 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":353,"threads_started":1}
I20260929 20:17:42.809998 6079 heartbeater.cc:347] Connected to a master server at 127.27.185.126:45975
I20260929 20:17:42.810357 6079 heartbeater.cc:464] Registering TS with master...
I20260929 20:17:42.810988 6079 heartbeater.cc:511] Master 127.27.185.126:45975 requested a full tablet report, sending...
I20260929 20:17:42.812635 5762 ts_manager.cc:194] Registered new tserver with Master: 34ef238932ad404ca6056e1394de3609 (127.27.185.66:35807)
W20260929 20:17:43.128826 6082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:43.129169 6082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:43.129362 6082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:43.136701 6082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:43.136858 6082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:43.137037 6082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:17:43.146651 6082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:45975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:43.148170 6082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:43.150060 6082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:43.163532 6090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:43.163645 6088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:43.163594 6087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:43.164528 6082 server_base.cc:1034] running on GCE node
I20260929 20:17:43.165203 6082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:43.166193 6082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:43.167484 6082 hybrid_clock.cc:648] HybridClock initialized: now 1790713063167453 us; error 72 us; skew 500 ppm
I20260929 20:17:43.167819 6082 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:17:43.170115 6082 webserver.cc:460] Webserver started at http://127.27.185.67:39161/ using document root <none> and password file <none>
I20260929 20:17:43.170835 6082 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:43.170969 6082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:43.171293 6082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:17:43.174194 6082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/data/instance:
uuid: "2c96c0755ba54173914f2e5698f9224c"
format_stamp: "Formatted at 2026-09-29 20:17:43 on dist-test-slave-4cx3"
server_key: "a12ac16f41f595f265d3db6fffb77f42"
server_key_iv: "95692413ba6137c6c2bbd4de24d915c8"
server_key_version: "encryptionkey@0"
I20260929 20:17:43.175220 6082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance:
uuid: "2c96c0755ba54173914f2e5698f9224c"
format_stamp: "Formatted at 2026-09-29 20:17:43 on dist-test-slave-4cx3"
server_key: "a12ac16f41f595f265d3db6fffb77f42"
server_key_iv: "95692413ba6137c6c2bbd4de24d915c8"
server_key_version: "encryptionkey@0"
I20260929 20:17:43.183801 6082 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.005s
I20260929 20:17:43.187655 6096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:43.189198 6082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:17:43.189437 6082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "2c96c0755ba54173914f2e5698f9224c"
format_stamp: "Formatted at 2026-09-29 20:17:43 on dist-test-slave-4cx3"
server_key: "a12ac16f41f595f265d3db6fffb77f42"
server_key_iv: "95692413ba6137c6c2bbd4de24d915c8"
server_key_version: "encryptionkey@0"
I20260929 20:17:43.189675 6082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:43.195524 6082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:43.196305 6082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:43.196576 6082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:43.197490 6082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:17:43.198977 6082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:17:43.199102 6082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:43.199223 6082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:17:43.199313 6082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:43.235354 6082 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:38705
I20260929 20:17:43.235373 6208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:38705 every 8 connection(s)
I20260929 20:17:43.236773 6082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:17:43.242424 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 6082
I20260929 20:17:43.242604 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance
I20260929 20:17:43.243219 28389 external_mini_cluster.cc:1428] Setting key 8b00eb456bdfbfd84ff9f145d59d5568
I20260929 20:17:43.247864 5950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:43.239867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:44153 (local address 127.27.185.126:45975)
0929 20:17:43.240090 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:43.240097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:43.242049 (+ 1952us) server_negotiation.cc:407] Connection header received
0929 20:17:43.245204 (+ 3155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:43.245211 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:43.245279 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:43.245431 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:43.247005 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:43.247011 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:43.247016 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:17:43.247062 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:43.247095 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:17:43.247107 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:43.247227 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:43.247470 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 20:17:43.247559 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 20:17:43.248957 6211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:43.240552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:45975 (local address 127.27.185.67:44153)
0929 20:17:43.241819 (+ 1267us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:43.241887 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:43.244778 (+ 2891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:43.245680 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:43.245689 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:43.245790 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:43.246691 (+ 901us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:17:43.246693 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:43.246734 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:17:43.246826 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:43.247205 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:43.247215 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:43.247222 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 20:17:43.247537 (+ 315us) client_negotiation.cc:240] Negotiation successful
0929 20:17:43.247811 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":306,"threads_started":1}
I20260929 20:17:43.250561 6209 heartbeater.cc:347] Connected to a master server at 127.27.185.126:45975
I20260929 20:17:43.250880 6209 heartbeater.cc:464] Registering TS with master...
I20260929 20:17:43.251571 6209 heartbeater.cc:511] Master 127.27.185.126:45975 requested a full tablet report, sending...
I20260929 20:17:43.253192 5762 ts_manager.cc:194] Registered new tserver with Master: 2c96c0755ba54173914f2e5698f9224c (127.27.185.67:38705)
I20260929 20:17:43.259249 28389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 20:17:43.267939 5950 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:17:43.266054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54696 (local address 127.27.185.126:45975)
0929 20:17:43.266302 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:43.266312 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:43.266890 (+ 578us) server_negotiation.cc:407] Connection header received
0929 20:17:43.267087 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:43.267091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:43.267147 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:43.267229 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:43.267675 (+ 446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54696: BlockingRecv error: recv got EOF from 127.0.0.1:54696 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:17:43.268723 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 5821
I20260929 20:17:43.273370 6218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:43.269801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56834 (local address 127.27.185.65:35811)
0929 20:17:43.270407 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:43.270420 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:43.270462 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:17:43.270549 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:43.270561 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:43.270924 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:43.271123 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:43.272336 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:43.272348 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:43.272367 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:17:43.272394 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:43.272458 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:17:43.272482 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:43.272677 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:43.272756 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 20:17:43.272884 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":223,"threads_started":1}
I20260929 20:17:43.274994 5943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:43.354900 5948 heartbeater.cc:503] Master 127.27.185.126:45975 was elected leader, sending a full tablet report...
I20260929 20:17:43.356335 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 5821
W20260929 20:17:43.372641 5746 connection.cc:535] server connection from 127.27.185.65:37301 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 20:17:43.372998 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 5952
I20260929 20:17:43.376996 6220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:43.374036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41374 (local address 127.27.185.66:35807)
0929 20:17:43.374777 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:43.374792 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:43.374840 (+ 48us) server_negotiation.cc:407] Connection header received
0929 20:17:43.374937 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:43.374944 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:43.375170 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:43.375307 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:43.376331 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:43.376339 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:43.376352 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:17:43.376377 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:43.376428 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:17:43.376444 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:43.376568 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:43.376652 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 20:17:43.376735 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":353,"threads_started":1}
I20260929 20:17:43.378290 6074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:43.458563 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 5952
I20260929 20:17:43.474472 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 6082
I20260929 20:17:43.478279 6222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:43.475452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36064 (local address 127.27.185.67:38705)
0929 20:17:43.476083 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:43.476090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:43.476120 (+ 30us) server_negotiation.cc:407] Connection header received
0929 20:17:43.476182 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:43.476188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:43.476423 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:43.476550 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:43.477561 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:43.477568 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:43.477580 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:17:43.477591 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:43.477631 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:17:43.477645 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:43.477778 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:43.477831 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 20:17:43.477946 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":326,"threads_started":1}
I20260929 20:17:43.479599 6204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:43.561129 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 6082
I20260929 20:17:43.577257 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 5730
I20260929 20:17:43.578589 5792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:43.681901 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 5730
2026-09-29T20:17:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2404 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:17:43 dist-test-slave-4cx3 krb5kdc[6227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:17:43 dist-test-slave-4cx3 krb5kdc[6227](info): set up 2 sockets
Sep 29 20:17:43 dist-test-slave-4cx3 krb5kdc[6227](info): commencing operation
krb5kdc: starting...
W20260929 20:17:44.795346 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.003s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:17:44 dist-test-slave-4cx3 krb5kdc[6227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:17:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:17:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
I20260929 20:17:45.031827 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:38481
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:33867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:38481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:17:45.365080 6240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:45.365406 6240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:45.365499 6240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:45.372795 6240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:17:45.372913 6240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:45.372961 6240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:45.373020 6240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:17:45.373101 6240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:17:45.383056 6240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:33867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:38481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:38481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:45.384500 6240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:45.385761 6240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:45.397354 6246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:45.397395 6248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:45.397357 6245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:45.398694 6240 server_base.cc:1034] running on GCE node
I20260929 20:17:45.399566 6240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:45.400991 6240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:45.402197 6240 hybrid_clock.cc:648] HybridClock initialized: now 1790713065402125 us; error 62 us; skew 500 ppm
I20260929 20:17:45.402518 6240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:17:45 dist-test-slave-4cx3 krb5kdc[6227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713065, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:17:45.405859 6240 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:17:45.407545 6240 webserver.cc:460] Webserver started at http://127.27.185.126:44825/ using document root <none> and password file <none>
I20260929 20:17:45.408255 6240 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:45.408409 6240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:45.408720 6240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:17:45.411842 6240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "d0bbd12e490e4f23aa64b2f0620683a8"
format_stamp: "Formatted at 2026-09-29 20:17:45 on dist-test-slave-4cx3"
server_key: "8efc539429c3bc33a55c9e015b3c58c8"
server_key_iv: "513ed67fc2b288af3fc55a37b3a66bb5"
server_key_version: "encryptionkey@0"
I20260929 20:17:45.412633 6240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "d0bbd12e490e4f23aa64b2f0620683a8"
format_stamp: "Formatted at 2026-09-29 20:17:45 on dist-test-slave-4cx3"
server_key: "8efc539429c3bc33a55c9e015b3c58c8"
server_key_iv: "513ed67fc2b288af3fc55a37b3a66bb5"
server_key_version: "encryptionkey@0"
I20260929 20:17:45.418344 6240 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260929 20:17:45.421929 6255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:45.423280 6240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 20:17:45.423529 6240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "d0bbd12e490e4f23aa64b2f0620683a8"
format_stamp: "Formatted at 2026-09-29 20:17:45 on dist-test-slave-4cx3"
server_key: "8efc539429c3bc33a55c9e015b3c58c8"
server_key_iv: "513ed67fc2b288af3fc55a37b3a66bb5"
server_key_version: "encryptionkey@0"
I20260929 20:17:45.423740 6240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:45.462016 6240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:45.466265 6240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:45.466564 6240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:45.486073 6240 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:38481
I20260929 20:17:45.486109 6306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:38481 every 8 connection(s)
I20260929 20:17:45.487563 6240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:17:45.491528 6307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:45.495679 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 6240
I20260929 20:17:45.495885 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:17:45.496556 28389 external_mini_cluster.cc:1428] Setting key a4d679be03e996198f76b42b711672e2
I20260929 20:17:45.503968 6307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0bbd12e490e4f23aa64b2f0620683a8: Bootstrap starting.
I20260929 20:17:45.507731 6307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0bbd12e490e4f23aa64b2f0620683a8: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:45.509071 6307 log.cc:824] T 00000000000000000000000000000000 P d0bbd12e490e4f23aa64b2f0620683a8: Log is configured to *not* fsync() on all Append() calls
I20260929 20:17:45.512373 6307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0bbd12e490e4f23aa64b2f0620683a8: No bootstrap required, opened a new log
Sep 29 20:17:45 dist-test-slave-4cx3 krb5kdc[6227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:17:45.522444 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:45.498546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60782 (local address 127.27.185.126:38481)
0929 20:17:45.499325 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:45.499345 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:45.499424 (+ 79us) server_negotiation.cc:407] Connection header received
0929 20:17:45.502178 (+ 2754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:45.502208 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:45.502710 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:45.503157 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:45.504609 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:45.506639 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:45.508177 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:45.509234 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:45.509426 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:45.513576 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:45.513587 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:45.513605 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:45.513649 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:45.517761 (+ 4112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:45.518475 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:45.518484 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:45.518493 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:45.518703 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:45.519070 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:45.519076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:45.519080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:45.519844 (+ 764us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:45.520160 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:45.520514 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 20:17:45.520979 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":316,"threads_started":1}
I20260929 20:17:45.523527 6307 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0bbd12e490e4f23aa64b2f0620683a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0bbd12e490e4f23aa64b2f0620683a8" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 38481 } }
I20260929 20:17:45.523910 6307 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0bbd12e490e4f23aa64b2f0620683a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:45.524053 6307 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0bbd12e490e4f23aa64b2f0620683a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0bbd12e490e4f23aa64b2f0620683a8, State: Initialized, Role: FOLLOWER
I20260929 20:17:45.524627 6307 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0bbd12e490e4f23aa64b2f0620683a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0bbd12e490e4f23aa64b2f0620683a8" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 38481 } }
I20260929 20:17:45.524884 6307 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0bbd12e490e4f23aa64b2f0620683a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:17:45.525039 6307 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0bbd12e490e4f23aa64b2f0620683a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:17:45.525228 6307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0bbd12e490e4f23aa64b2f0620683a8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:17:45.527832 6307 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0bbd12e490e4f23aa64b2f0620683a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0bbd12e490e4f23aa64b2f0620683a8" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 38481 } }
I20260929 20:17:45.528349 6307 leader_election.cc:302] T 00000000000000000000000000000000 P d0bbd12e490e4f23aa64b2f0620683a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0bbd12e490e4f23aa64b2f0620683a8; no voters:
I20260929 20:17:45.528951 6307 leader_election.cc:288] T 00000000000000000000000000000000 P d0bbd12e490e4f23aa64b2f0620683a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:17:45.529104 6312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0bbd12e490e4f23aa64b2f0620683a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:17:45.529944 6312 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0bbd12e490e4f23aa64b2f0620683a8 [term 1 LEADER]: Becoming Leader. State: Replica: d0bbd12e490e4f23aa64b2f0620683a8, State: Running, Role: LEADER
I20260929 20:17:45.530437 6312 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0bbd12e490e4f23aa64b2f0620683a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0bbd12e490e4f23aa64b2f0620683a8" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 38481 } }
I20260929 20:17:45.531201 6307 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0bbd12e490e4f23aa64b2f0620683a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:17:45.533475 6313 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0bbd12e490e4f23aa64b2f0620683a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0bbd12e490e4f23aa64b2f0620683a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0bbd12e490e4f23aa64b2f0620683a8" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 38481 } } }
I20260929 20:17:45.533846 6313 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0bbd12e490e4f23aa64b2f0620683a8 [sys.catalog]: This master's current role is: LEADER
I20260929 20:17:45.533587 6314 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0bbd12e490e4f23aa64b2f0620683a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0bbd12e490e4f23aa64b2f0620683a8. Latest consensus state: current_term: 1 leader_uuid: "d0bbd12e490e4f23aa64b2f0620683a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0bbd12e490e4f23aa64b2f0620683a8" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 38481 } } }
I20260929 20:17:45.534071 6314 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0bbd12e490e4f23aa64b2f0620683a8 [sys.catalog]: This master's current role is: LEADER
I20260929 20:17:45.536279 6318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:17:45.541723 6318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:17:45.551489 6318 catalog_manager.cc:1347] Generated new cluster ID: a15275e48a6d42209c9f418eae491ba5
I20260929 20:17:45.551631 6318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:17:45.571627 6318 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:17:45.573366 6318 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:17:45.584280 6318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0bbd12e490e4f23aa64b2f0620683a8: Generated new TSK 0
I20260929 20:17:45.585366 6318 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
I20260929 20:17:45.659811 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:0
--local_ip_for_outbound_sockets=127.27.185.65
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:38481
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:33867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:17:45.994185 6333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:45.994488 6333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:45.994654 6333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:46.002043 6333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:46.002156 6333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:46.002336 6333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:17:46.012388 6333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:33867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:38481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:46.013760 6333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:46.015097 6333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:46.029155 6339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:46.029268 6338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:46.029215 6341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:46.033140 6333 server_base.cc:1034] running on GCE node
I20260929 20:17:46.033808 6333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:46.034837 6333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:46.036079 6333 hybrid_clock.cc:648] HybridClock initialized: now 1790713066036027 us; error 46 us; skew 500 ppm
I20260929 20:17:46.036415 6333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:17:46 dist-test-slave-4cx3 krb5kdc[6227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713066, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:17:46.039981 6333 init.cc:377] Logged in from keytab as kudu/127.27.185.65@KRBTEST.COM (short username kudu)
I20260929 20:17:46.041603 6333 webserver.cc:460] Webserver started at http://127.27.185.65:45675/ using document root <none> and password file <none>
I20260929 20:17:46.042294 6333 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:46.042402 6333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:46.042639 6333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:17:46.045641 6333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/data/instance:
uuid: "a5399ce3a7ce41edb1b6a9223be44697"
format_stamp: "Formatted at 2026-09-29 20:17:46 on dist-test-slave-4cx3"
server_key: "5e630ae16690fd728321cabd52ebb245"
server_key_iv: "9816713964e41837e9940d99b959c909"
server_key_version: "encryptionkey@0"
I20260929 20:17:46.046341 6333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance:
uuid: "a5399ce3a7ce41edb1b6a9223be44697"
format_stamp: "Formatted at 2026-09-29 20:17:46 on dist-test-slave-4cx3"
server_key: "5e630ae16690fd728321cabd52ebb245"
server_key_iv: "9816713964e41837e9940d99b959c909"
server_key_version: "encryptionkey@0"
I20260929 20:17:46.051661 6333 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 20:17:46.055085 6348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:46.056632 6333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:17:46.056829 6333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "a5399ce3a7ce41edb1b6a9223be44697"
format_stamp: "Formatted at 2026-09-29 20:17:46 on dist-test-slave-4cx3"
server_key: "5e630ae16690fd728321cabd52ebb245"
server_key_iv: "9816713964e41837e9940d99b959c909"
server_key_version: "encryptionkey@0"
I20260929 20:17:46.057030 6333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:46.088629 6333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:46.092729 6333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:46.093026 6333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:46.093966 6333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:17:46.095309 6333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:17:46.095420 6333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:46.095587 6333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:17:46.095672 6333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:46.129029 6333 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:39401
I20260929 20:17:46.129072 6460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:39401 every 8 connection(s)
I20260929 20:17:46.130370 6333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:17:46.133216 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 6333
I20260929 20:17:46.133438 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance
I20260929 20:17:46.134495 28389 external_mini_cluster.cc:1428] Setting key 744920cb4cbad758a90be09778c1986f
Sep 29 20:17:46 dist-test-slave-4cx3 krb5kdc[6227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713066, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:17:46.152357 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:46.133177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:52735 (local address 127.27.185.126:38481)
0929 20:17:46.135688 (+ 2511us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:46.135693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:46.135730 (+ 37us) server_negotiation.cc:407] Connection header received
0929 20:17:46.137065 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:46.137070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:46.137153 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:46.137279 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:46.138768 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:46.139987 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:46.141575 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:46.142313 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:46.142461 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:46.146191 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:46.146197 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:46.146200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:46.146231 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:46.148865 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:46.149876 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:46.149880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:46.149882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:46.150007 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:46.150589 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:46.150593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:46.150595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:46.150855 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:46.150984 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:46.151920 (+ 936us) server_negotiation.cc:299] Negotiation successful
0929 20:17:46.152094 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2406,"thread_start_us":2253,"threads_started":1}
I20260929 20:17:46.153970 6464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:46.133745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:38481 (local address 127.27.185.65:52735)
0929 20:17:46.134583 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:46.134636 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:46.136805 (+ 2169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:46.137506 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:46.137515 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:46.138021 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:46.138589 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:46.138607 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:46.140137 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:46.140141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:46.141407 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:46.141419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:46.142548 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:46.142553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:46.142746 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:46.143492 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:46.143515 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:46.145970 (+ 2455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:46.149034 (+ 3064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:46.149046 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:46.149067 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:46.149762 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:46.150206 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:46.150213 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:46.150220 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:46.150479 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:46.150993 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:46.151004 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:46.151499 (+ 495us) client_negotiation.cc:769] Sending connection context
0929 20:17:46.151808 (+ 309us) client_negotiation.cc:240] Negotiation successful
0929 20:17:46.152432 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":361,"threads_started":1}
I20260929 20:17:46.155699 6461 heartbeater.cc:347] Connected to a master server at 127.27.185.126:38481
I20260929 20:17:46.156026 6461 heartbeater.cc:464] Registering TS with master...
I20260929 20:17:46.156772 6461 heartbeater.cc:511] Master 127.27.185.126:38481 requested a full tablet report, sending...
I20260929 20:17:46.158901 6272 ts_manager.cc:194] Registered new tserver with Master: a5399ce3a7ce41edb1b6a9223be44697 (127.27.185.65:39401)
WARNING: no policy specified for kudu/127.27.185.66@KRBTEST.COM; defaulting to no policy
I20260929 20:17:46.160709 6272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.65@KRBTEST.COM'} at 127.27.185.65:52735
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
I20260929 20:17:46.191489 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:0
--local_ip_for_outbound_sockets=127.27.185.66
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:38481
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:33867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:17:46.529839 6467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:46.530141 6467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:46.530296 6467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:46.537983 6467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:46.538098 6467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:46.538308 6467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:17:46.548426 6467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:33867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:38481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:46.549831 6467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:46.551268 6467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:46.564957 6472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:46.564980 6473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:46.565675 6475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:46.565899 6467 server_base.cc:1034] running on GCE node
I20260929 20:17:46.566627 6467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:46.567808 6467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:46.569098 6467 hybrid_clock.cc:648] HybridClock initialized: now 1790713066569009 us; error 64 us; skew 500 ppm
I20260929 20:17:46.569446 6467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:17:46 dist-test-slave-4cx3 krb5kdc[6227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713066, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:17:46.572928 6467 init.cc:377] Logged in from keytab as kudu/127.27.185.66@KRBTEST.COM (short username kudu)
I20260929 20:17:46.574532 6467 webserver.cc:460] Webserver started at http://127.27.185.66:34771/ using document root <none> and password file <none>
I20260929 20:17:46.575237 6467 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:46.575338 6467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:46.575673 6467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:17:46.578740 6467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/data/instance:
uuid: "ad8e09f9bd4641df83c9aba7c379d4ed"
format_stamp: "Formatted at 2026-09-29 20:17:46 on dist-test-slave-4cx3"
server_key: "fd90888316f3d0b9da6d10958320c303"
server_key_iv: "43b930c5814a34ed05ced019bdfcce46"
server_key_version: "encryptionkey@0"
I20260929 20:17:46.579622 6467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance:
uuid: "ad8e09f9bd4641df83c9aba7c379d4ed"
format_stamp: "Formatted at 2026-09-29 20:17:46 on dist-test-slave-4cx3"
server_key: "fd90888316f3d0b9da6d10958320c303"
server_key_iv: "43b930c5814a34ed05ced019bdfcce46"
server_key_version: "encryptionkey@0"
I20260929 20:17:46.585150 6467 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 20:17:46.588685 6482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:46.590117 6467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:17:46.590255 6467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "ad8e09f9bd4641df83c9aba7c379d4ed"
format_stamp: "Formatted at 2026-09-29 20:17:46 on dist-test-slave-4cx3"
server_key: "fd90888316f3d0b9da6d10958320c303"
server_key_iv: "43b930c5814a34ed05ced019bdfcce46"
server_key_version: "encryptionkey@0"
I20260929 20:17:46.590425 6467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:46.613086 6467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:46.617375 6467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:46.617686 6467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:46.618700 6467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:17:46.620081 6467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:17:46.620158 6467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:46.620291 6467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:17:46.620342 6467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:46.654036 6467 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:43569
I20260929 20:17:46.654207 6594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:43569 every 8 connection(s)
I20260929 20:17:46.655491 6467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:17:46.664495 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 6467
I20260929 20:17:46.664718 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance
I20260929 20:17:46.665174 28389 external_mini_cluster.cc:1428] Setting key d7baa2a93cd9fa93f0473abfa90ae929
Sep 29 20:17:46 dist-test-slave-4cx3 krb5kdc[6227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713066, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:17:46.678273 6597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:46.658217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:55629 (local address 127.27.185.126:38481)
0929 20:17:46.658692 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:46.658699 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:46.659987 (+ 1288us) server_negotiation.cc:407] Connection header received
0929 20:17:46.662092 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:46.662096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:46.662170 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:46.662277 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:46.663904 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:46.664715 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:46.666699 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:46.667304 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:46.667464 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:46.671182 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:46.671191 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:46.671199 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:46.671246 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:46.674876 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:46.675812 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:46.675816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:46.675818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:46.675953 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:46.676520 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:46.676523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:46.676525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:46.676779 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:46.676888 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:46.677810 (+ 922us) server_negotiation.cc:299] Negotiation successful
0929 20:17:46.677985 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":245,"threads_started":1}
I20260929 20:17:46.679874 6598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:46.658844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:38481 (local address 127.27.185.66:55629)
0929 20:17:46.659784 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:46.659838 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:46.661807 (+ 1969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:46.662483 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:46.662491 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:46.663032 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:46.663704 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:46.663728 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:46.664893 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:46.664903 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:46.666555 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:46.666569 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:46.667569 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:46.667574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:46.667780 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:46.668509 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:46.668533 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:46.670929 (+ 2396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:46.675028 (+ 4099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:46.675035 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:46.675050 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:46.675685 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:46.676071 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:46.676078 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:46.676083 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:46.676347 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:46.676907 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:46.676919 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:46.677405 (+ 486us) client_negotiation.cc:769] Sending connection context
0929 20:17:46.677713 (+ 308us) client_negotiation.cc:240] Negotiation successful
0929 20:17:46.678332 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":314,"threads_started":1}
I20260929 20:17:46.681481 6595 heartbeater.cc:347] Connected to a master server at 127.27.185.126:38481
I20260929 20:17:46.681854 6595 heartbeater.cc:464] Registering TS with master...
I20260929 20:17:46.682667 6595 heartbeater.cc:511] Master 127.27.185.126:38481 requested a full tablet report, sending...
I20260929 20:17:46.684386 6272 ts_manager.cc:194] Registered new tserver with Master: ad8e09f9bd4641df83c9aba7c379d4ed (127.27.185.66:43569)
I20260929 20:17:46.685283 6272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.66@KRBTEST.COM'} at 127.27.185.66:55629
WARNING: no policy specified for kudu/127.27.185.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
I20260929 20:17:46.722795 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:0
--local_ip_for_outbound_sockets=127.27.185.67
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:38481
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:33867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:17:47.054674 6601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:17:47.055064 6601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:17:47.055249 6601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:17:47.062727 6601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:17:47.062844 6601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:17:47.063071 6601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:17:47.072970 6601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:33867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:38481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:17:47.074317 6601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:17:47.075839 6601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:17:47.090035 6606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:47.090035 6607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:17:47.090035 6609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:17:47.091012 6601 server_base.cc:1034] running on GCE node
I20260929 20:17:47.091717 6601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:17:47.092742 6601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:17:47.093988 6601 hybrid_clock.cc:648] HybridClock initialized: now 1790713067093939 us; error 50 us; skew 500 ppm
I20260929 20:17:47.094347 6601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:17:47 dist-test-slave-4cx3 krb5kdc[6227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713067, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:17:47.097730 6601 init.cc:377] Logged in from keytab as kudu/127.27.185.67@KRBTEST.COM (short username kudu)
I20260929 20:17:47.099256 6601 webserver.cc:460] Webserver started at http://127.27.185.67:40011/ using document root <none> and password file <none>
I20260929 20:17:47.100036 6601 fs_manager.cc:359] Metadata directory not provided
I20260929 20:17:47.100185 6601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:17:47.100499 6601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:17:47.104087 6601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/data/instance:
uuid: "4c9504082893400f92cd853b64a29061"
format_stamp: "Formatted at 2026-09-29 20:17:47 on dist-test-slave-4cx3"
server_key: "670f6e3e61416c981d086c65d04063ed"
server_key_iv: "f227be6ee9973192fe044245c180886b"
server_key_version: "encryptionkey@0"
I20260929 20:17:47.104853 6601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance:
uuid: "4c9504082893400f92cd853b64a29061"
format_stamp: "Formatted at 2026-09-29 20:17:47 on dist-test-slave-4cx3"
server_key: "670f6e3e61416c981d086c65d04063ed"
server_key_iv: "f227be6ee9973192fe044245c180886b"
server_key_version: "encryptionkey@0"
I20260929 20:17:47.110122 6601 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:17:47.113517 6616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:47.114814 6601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:17:47.115020 6601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "4c9504082893400f92cd853b64a29061"
format_stamp: "Formatted at 2026-09-29 20:17:47 on dist-test-slave-4cx3"
server_key: "670f6e3e61416c981d086c65d04063ed"
server_key_iv: "f227be6ee9973192fe044245c180886b"
server_key_version: "encryptionkey@0"
I20260929 20:17:47.115221 6601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:17:47.141098 6601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:17:47.145121 6601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:17:47.145430 6601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:17:47.146472 6601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:17:47.147881 6601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:17:47.147995 6601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:47.148146 6601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:17:47.148219 6601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:17:47.164386 6461 heartbeater.cc:503] Master 127.27.185.126:38481 was elected leader, sending a full tablet report...
I20260929 20:17:47.182611 6601 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:40741
I20260929 20:17:47.182657 6728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:40741 every 8 connection(s)
I20260929 20:17:47.183938 6601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:17:47.187495 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 6601
I20260929 20:17:47.187716 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance
I20260929 20:17:47.188261 28389 external_mini_cluster.cc:1428] Setting key 4d2544144b6b46b23722464ffa6a49c7
Sep 29 20:17:47 dist-test-slave-4cx3 krb5kdc[6227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713067, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:17:47.204604 6731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:47.186553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:46905 (local address 127.27.185.126:38481)
0929 20:17:47.186995 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:47.187000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:47.188222 (+ 1222us) server_negotiation.cc:407] Connection header received
0929 20:17:47.190428 (+ 2206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:47.190434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:47.190546 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:47.190664 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:47.192338 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.193125 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.194667 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.195065 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.195193 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:47.198510 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:47.198515 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:47.198518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:47.198557 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:47.201119 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:47.201841 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:47.201851 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:47.201853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:47.201978 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:47.202473 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:47.202480 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:47.202486 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:47.202800 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:47.202943 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:47.203970 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0929 20:17:47.204208 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":216,"threads_started":1}
I20260929 20:17:47.205622 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:47.187192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:38481 (local address 127.27.185.67:46905)
0929 20:17:47.188066 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:47.188111 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:47.190148 (+ 2037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:47.190899 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:47.190909 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:47.191606 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:17:47.192173 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:47.192191 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.193302 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.193307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:47.194521 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:47.194537 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.195319 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.195328 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:47.195599 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:47.196196 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:17:47.196218 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:17:47.198299 (+ 2081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:17:47.201259 (+ 2960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:47.201268 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:47.201284 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:47.201709 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:47.202091 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:17:47.202096 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:17:47.202100 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:17:47.202324 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:17:47.203049 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:17:47.203058 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:17:47.203535 (+ 477us) client_negotiation.cc:769] Sending connection context
0929 20:17:47.203873 (+ 338us) client_negotiation.cc:240] Negotiation successful
0929 20:17:47.204422 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":383,"threads_started":1}
I20260929 20:17:47.206944 6729 heartbeater.cc:347] Connected to a master server at 127.27.185.126:38481
I20260929 20:17:47.207257 6729 heartbeater.cc:464] Registering TS with master...
I20260929 20:17:47.208038 6729 heartbeater.cc:511] Master 127.27.185.126:38481 requested a full tablet report, sending...
I20260929 20:17:47.209442 6272 ts_manager.cc:194] Registered new tserver with Master: 4c9504082893400f92cd853b64a29061 (127.27.185.67:40741)
I20260929 20:17:47.210300 6272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.67@KRBTEST.COM'} at 127.27.185.67:46905
I20260929 20:17:47.216490 28389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:17:47.233778 6731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:47.221157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60786 (local address 127.27.185.126:38481)
0929 20:17:47.221404 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:47.221422 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:47.222023 (+ 601us) server_negotiation.cc:407] Connection header received
0929 20:17:47.222093 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:47.222098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:47.222186 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:47.222267 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:47.223848 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.224585 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.225824 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.226227 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.226360 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:47.228713 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:47.228719 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:47.228722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:47.228754 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:47.231284 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:47.231949 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:47.231954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:47.231956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:47.232069 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:47.232520 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:47.232527 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:47.232530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:47.232910 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:47.233087 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:47.233383 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 20:17:47.233551 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:17:47.237780 6272 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:17:47.246058 6272 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:17:47.279997 6744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:47.259316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57882 (local address 127.27.185.67:40741)
0929 20:17:47.271371 (+ 12055us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:47.271400 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:47.272077 (+ 677us) server_negotiation.cc:407] Connection header received
0929 20:17:47.272191 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:47.272201 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:47.272452 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:47.273388 (+ 936us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:17:47.273648 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.275165 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.277734 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.278629 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.279093 (+ 464us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:47.279376 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:47.279512 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 20:17:47.279674 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11866,"thread_start_us":11695,"threads_started":1}
I20260929 20:17:47.280071 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:47.259101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:40741 (local address 127.0.0.1:57882)
0929 20:17:47.271780 (+ 12679us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:47.271801 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:47.271933 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:47.272667 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:47.272671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:47.272700 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:17:47.273188 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:47.273201 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.275317 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.275322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:47.277577 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:47.277589 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.278877 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.278883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:47.279212 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:47.279306 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 20:17:47.279477 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 20:17:47.279665 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12540,"mutex_wait_us":320,"thread_start_us":12196,"threads_started":1}
I20260929 20:17:47.281505 6747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:47.272758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:43569 (local address 127.0.0.1:56906)
0929 20:17:47.273397 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:47.273422 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:47.273615 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:47.274383 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:47.274388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:47.274423 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:17:47.274788 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:47.274796 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.276127 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.276136 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:47.279135 (+ 2999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:47.279151 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.280746 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.280750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:47.280874 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:47.280895 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 20:17:47.281008 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 20:17:47.281130 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":66,"thread_start_us":359,"threads_started":1}
I20260929 20:17:47.281541 6746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:47.272170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56906 (local address 127.27.185.66:43569)
0929 20:17:47.272676 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:47.272684 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:47.273520 (+ 836us) server_negotiation.cc:407] Connection header received
0929 20:17:47.273801 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:47.273809 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:47.274025 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:47.274192 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:17:47.274947 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.275988 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.279326 (+ 3338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.280533 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.280703 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:47.280878 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:47.281076 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 20:17:47.281243 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":294,"threads_started":1}
I20260929 20:17:47.282203 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:47.259098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:39401 (local address 127.0.0.1:53216)
0929 20:17:47.270568 (+ 11470us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:47.270607 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:47.270823 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:47.271813 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:47.271820 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:47.271853 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:17:47.272254 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:47.272266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.274259 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.274265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:47.276892 (+ 2627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:47.276908 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.278855 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.278863 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:47.279228 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:47.281742 (+ 2514us) client_negotiation.cc:769] Sending connection context
0929 20:17:47.281832 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 20:17:47.281948 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11221,"mutex_wait_us":2530,"thread_start_us":11092,"threads_started":1}
I20260929 20:17:47.282395 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:47.259580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53216 (local address 127.27.185.65:39401)
0929 20:17:47.270170 (+ 10590us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:47.270182 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:47.271066 (+ 884us) server_negotiation.cc:407] Connection header received
0929 20:17:47.271278 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:47.271287 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:47.271578 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:47.271785 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:17:47.272488 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.274109 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.277066 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.278600 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.278734 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:47.278987 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:47.281894 (+ 2907us) server_negotiation.cc:299] Negotiation successful
0929 20:17:47.282070 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10381,"thread_start_us":10202,"threads_started":1}
I20260929 20:17:47.284829 6664 tablet_service.cc:1467] Processing CreateTablet for tablet 7fb2a464ae384e0191ac337addcb1ef2 (DEFAULT_TABLE table=test-table [id=1f182af9dedb46e09fd1c374ba08792e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:17:47.286415 6664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fb2a464ae384e0191ac337addcb1ef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:47.286075 6396 tablet_service.cc:1467] Processing CreateTablet for tablet 7fb2a464ae384e0191ac337addcb1ef2 (DEFAULT_TABLE table=test-table [id=1f182af9dedb46e09fd1c374ba08792e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:17:47.286670 6530 tablet_service.cc:1467] Processing CreateTablet for tablet 7fb2a464ae384e0191ac337addcb1ef2 (DEFAULT_TABLE table=test-table [id=1f182af9dedb46e09fd1c374ba08792e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:17:47.287456 6396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fb2a464ae384e0191ac337addcb1ef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:47.287933 6530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fb2a464ae384e0191ac337addcb1ef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:17:47.300937 6748 tablet_bootstrap.cc:492] T 7fb2a464ae384e0191ac337addcb1ef2 P ad8e09f9bd4641df83c9aba7c379d4ed: Bootstrap starting.
I20260929 20:17:47.304198 6748 tablet_bootstrap.cc:654] T 7fb2a464ae384e0191ac337addcb1ef2 P ad8e09f9bd4641df83c9aba7c379d4ed: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:47.304698 6750 tablet_bootstrap.cc:492] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697: Bootstrap starting.
I20260929 20:17:47.305192 6749 tablet_bootstrap.cc:492] T 7fb2a464ae384e0191ac337addcb1ef2 P 4c9504082893400f92cd853b64a29061: Bootstrap starting.
I20260929 20:17:47.305590 6748 log.cc:824] T 7fb2a464ae384e0191ac337addcb1ef2 P ad8e09f9bd4641df83c9aba7c379d4ed: Log is configured to *not* fsync() on all Append() calls
I20260929 20:17:47.308182 6750 tablet_bootstrap.cc:654] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:47.309069 6748 tablet_bootstrap.cc:492] T 7fb2a464ae384e0191ac337addcb1ef2 P ad8e09f9bd4641df83c9aba7c379d4ed: No bootstrap required, opened a new log
I20260929 20:17:47.309104 6750 log.cc:824] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697: Log is configured to *not* fsync() on all Append() calls
I20260929 20:17:47.309353 6748 ts_tablet_manager.cc:1397] T 7fb2a464ae384e0191ac337addcb1ef2 P ad8e09f9bd4641df83c9aba7c379d4ed: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260929 20:17:47.311707 6750 tablet_bootstrap.cc:492] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697: No bootstrap required, opened a new log
I20260929 20:17:47.311964 6750 ts_tablet_manager.cc:1397] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 20:17:47.313547 6749 tablet_bootstrap.cc:654] T 7fb2a464ae384e0191ac337addcb1ef2 P 4c9504082893400f92cd853b64a29061: Neither blocks nor log segments found. Creating new log.
I20260929 20:17:47.314769 6749 log.cc:824] T 7fb2a464ae384e0191ac337addcb1ef2 P 4c9504082893400f92cd853b64a29061: Log is configured to *not* fsync() on all Append() calls
I20260929 20:17:47.318437 6749 tablet_bootstrap.cc:492] T 7fb2a464ae384e0191ac337addcb1ef2 P 4c9504082893400f92cd853b64a29061: No bootstrap required, opened a new log
I20260929 20:17:47.318712 6749 ts_tablet_manager.cc:1397] T 7fb2a464ae384e0191ac337addcb1ef2 P 4c9504082893400f92cd853b64a29061: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.001s
I20260929 20:17:47.318945 6748 raft_consensus.cc:348] T 7fb2a464ae384e0191ac337addcb1ef2 P ad8e09f9bd4641df83c9aba7c379d4ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9504082893400f92cd853b64a29061" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40741 } } peers { permanent_uuid: "a5399ce3a7ce41edb1b6a9223be44697" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 39401 } } peers { permanent_uuid: "ad8e09f9bd4641df83c9aba7c379d4ed" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43569 } }
I20260929 20:17:47.319490 6748 raft_consensus.cc:374] T 7fb2a464ae384e0191ac337addcb1ef2 P ad8e09f9bd4641df83c9aba7c379d4ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:47.319607 6748 raft_consensus.cc:729] T 7fb2a464ae384e0191ac337addcb1ef2 P ad8e09f9bd4641df83c9aba7c379d4ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad8e09f9bd4641df83c9aba7c379d4ed, State: Initialized, Role: FOLLOWER
I20260929 20:17:47.320312 6750 raft_consensus.cc:348] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9504082893400f92cd853b64a29061" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40741 } } peers { permanent_uuid: "a5399ce3a7ce41edb1b6a9223be44697" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 39401 } } peers { permanent_uuid: "ad8e09f9bd4641df83c9aba7c379d4ed" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43569 } }
I20260929 20:17:47.320328 6748 consensus_queue.cc:260] T 7fb2a464ae384e0191ac337addcb1ef2 P ad8e09f9bd4641df83c9aba7c379d4ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9504082893400f92cd853b64a29061" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40741 } } peers { permanent_uuid: "a5399ce3a7ce41edb1b6a9223be44697" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 39401 } } peers { permanent_uuid: "ad8e09f9bd4641df83c9aba7c379d4ed" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43569 } }
I20260929 20:17:47.320643 6750 raft_consensus.cc:374] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:47.320760 6750 raft_consensus.cc:729] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5399ce3a7ce41edb1b6a9223be44697, State: Initialized, Role: FOLLOWER
I20260929 20:17:47.321457 6595 heartbeater.cc:503] Master 127.27.185.126:38481 was elected leader, sending a full tablet report...
I20260929 20:17:47.321498 6750 consensus_queue.cc:260] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9504082893400f92cd853b64a29061" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40741 } } peers { permanent_uuid: "a5399ce3a7ce41edb1b6a9223be44697" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 39401 } } peers { permanent_uuid: "ad8e09f9bd4641df83c9aba7c379d4ed" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43569 } }
I20260929 20:17:47.322095 6748 ts_tablet_manager.cc:1428] T 7fb2a464ae384e0191ac337addcb1ef2 P ad8e09f9bd4641df83c9aba7c379d4ed: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260929 20:17:47.323688 6750 ts_tablet_manager.cc:1428] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260929 20:17:47.324579 6755 raft_consensus.cc:482] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:17:47.324870 6755 raft_consensus.cc:504] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9504082893400f92cd853b64a29061" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40741 } } peers { permanent_uuid: "a5399ce3a7ce41edb1b6a9223be44697" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 39401 } } peers { permanent_uuid: "ad8e09f9bd4641df83c9aba7c379d4ed" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43569 } }
I20260929 20:17:47.331089 6749 raft_consensus.cc:348] T 7fb2a464ae384e0191ac337addcb1ef2 P 4c9504082893400f92cd853b64a29061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9504082893400f92cd853b64a29061" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40741 } } peers { permanent_uuid: "a5399ce3a7ce41edb1b6a9223be44697" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 39401 } } peers { permanent_uuid: "ad8e09f9bd4641df83c9aba7c379d4ed" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43569 } }
I20260929 20:17:47.331736 6749 raft_consensus.cc:374] T 7fb2a464ae384e0191ac337addcb1ef2 P 4c9504082893400f92cd853b64a29061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:17:47.331882 6749 raft_consensus.cc:729] T 7fb2a464ae384e0191ac337addcb1ef2 P 4c9504082893400f92cd853b64a29061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c9504082893400f92cd853b64a29061, State: Initialized, Role: FOLLOWER
I20260929 20:17:47.332690 6749 consensus_queue.cc:260] T 7fb2a464ae384e0191ac337addcb1ef2 P 4c9504082893400f92cd853b64a29061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9504082893400f92cd853b64a29061" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40741 } } peers { permanent_uuid: "a5399ce3a7ce41edb1b6a9223be44697" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 39401 } } peers { permanent_uuid: "ad8e09f9bd4641df83c9aba7c379d4ed" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43569 } }
I20260929 20:17:47.333319 6755 leader_election.cc:288] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c9504082893400f92cd853b64a29061 (127.27.185.67:40741), ad8e09f9bd4641df83c9aba7c379d4ed (127.27.185.66:43569)
I20260929 20:17:47.333848 6729 heartbeater.cc:503] Master 127.27.185.126:38481 was elected leader, sending a full tablet report...
I20260929 20:17:47.334837 6749 ts_tablet_manager.cc:1428] T 7fb2a464ae384e0191ac337addcb1ef2 P 4c9504082893400f92cd853b64a29061: Time spent starting tablet: real 0.016s user 0.009s sys 0.006s
I20260929 20:17:47.342504 6744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:47.327669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:49641 (local address 127.27.185.67:40741)
0929 20:17:47.327894 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:47.327900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:47.335833 (+ 7933us) server_negotiation.cc:407] Connection header received
0929 20:17:47.336050 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:47.336056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:47.336128 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:47.336231 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:17:47.336974 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.338095 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.340527 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.341525 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.341686 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:47.341757 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:47.342110 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 20:17:47.342236 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:17:47.342612 6756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:47.327105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:40741 (local address 127.27.185.65:49641)
0929 20:17:47.335713 (+ 8608us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:47.335735 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:47.335875 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:47.336268 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:47.336273 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:47.336303 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:17:47.336744 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:47.336769 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.338277 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.338282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:47.340357 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:47.340372 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.341749 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.341757 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:47.341951 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:47.341986 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 20:17:47.342084 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 20:17:47.342217 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8466,"mutex_wait_us":1,"thread_start_us":8238,"threads_started":1}
I20260929 20:17:47.343559 6746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:47.336158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:45077 (local address 127.27.185.66:43569)
0929 20:17:47.336363 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:47.336369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:47.337309 (+ 940us) server_negotiation.cc:407] Connection header received
0929 20:17:47.337483 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:47.337488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:47.337556 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:47.337644 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:17:47.338317 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.339048 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.341147 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.342379 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.342545 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:47.342610 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:47.343194 (+ 584us) server_negotiation.cc:299] Negotiation successful
0929 20:17:47.343308 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 20:17:47.343652 6758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:47.336530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:43569 (local address 127.27.185.65:45077)
0929 20:17:47.337187 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0929 20:17:47.337205 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:17:47.337345 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:17:47.337797 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:17:47.337801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:17:47.337825 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:17:47.338165 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:47.338177 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.339189 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.339195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:47.340982 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:17:47.340993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.342906 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.342911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:17:47.343030 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:47.343052 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 20:17:47.343130 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 20:17:47.343262 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":313,"threads_started":1}
I20260929 20:17:47.343750 6684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fb2a464ae384e0191ac337addcb1ef2" candidate_uuid: "a5399ce3a7ce41edb1b6a9223be44697" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c9504082893400f92cd853b64a29061" is_pre_election: true
I20260929 20:17:47.344372 6684 raft_consensus.cc:2454] T 7fb2a464ae384e0191ac337addcb1ef2 P 4c9504082893400f92cd853b64a29061 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5399ce3a7ce41edb1b6a9223be44697 in term 0.
I20260929 20:17:47.344756 6550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fb2a464ae384e0191ac337addcb1ef2" candidate_uuid: "a5399ce3a7ce41edb1b6a9223be44697" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad8e09f9bd4641df83c9aba7c379d4ed" is_pre_election: true
I20260929 20:17:47.345115 6349 leader_election.cc:302] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c9504082893400f92cd853b64a29061, a5399ce3a7ce41edb1b6a9223be44697; no voters:
I20260929 20:17:47.345201 6550 raft_consensus.cc:2454] T 7fb2a464ae384e0191ac337addcb1ef2 P ad8e09f9bd4641df83c9aba7c379d4ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5399ce3a7ce41edb1b6a9223be44697 in term 0.
I20260929 20:17:47.345530 6755 raft_consensus.cc:2781] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:17:47.345703 6755 raft_consensus.cc:482] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:17:47.345808 6755 raft_consensus.cc:3037] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:17:47.348150 6755 raft_consensus.cc:504] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9504082893400f92cd853b64a29061" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40741 } } peers { permanent_uuid: "a5399ce3a7ce41edb1b6a9223be44697" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 39401 } } peers { permanent_uuid: "ad8e09f9bd4641df83c9aba7c379d4ed" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43569 } }
I20260929 20:17:47.348719 6755 leader_election.cc:288] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697 [CANDIDATE]: Term 1 election: Requested vote from peers 4c9504082893400f92cd853b64a29061 (127.27.185.67:40741), ad8e09f9bd4641df83c9aba7c379d4ed (127.27.185.66:43569)
I20260929 20:17:47.349565 6550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fb2a464ae384e0191ac337addcb1ef2" candidate_uuid: "a5399ce3a7ce41edb1b6a9223be44697" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad8e09f9bd4641df83c9aba7c379d4ed"
I20260929 20:17:47.349627 6684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fb2a464ae384e0191ac337addcb1ef2" candidate_uuid: "a5399ce3a7ce41edb1b6a9223be44697" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c9504082893400f92cd853b64a29061"
I20260929 20:17:47.349815 6684 raft_consensus.cc:3037] T 7fb2a464ae384e0191ac337addcb1ef2 P 4c9504082893400f92cd853b64a29061 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:17:47.349817 6550 raft_consensus.cc:3037] T 7fb2a464ae384e0191ac337addcb1ef2 P ad8e09f9bd4641df83c9aba7c379d4ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:17:47.353116 6684 raft_consensus.cc:2454] T 7fb2a464ae384e0191ac337addcb1ef2 P 4c9504082893400f92cd853b64a29061 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5399ce3a7ce41edb1b6a9223be44697 in term 1.
I20260929 20:17:47.353139 6550 raft_consensus.cc:2454] T 7fb2a464ae384e0191ac337addcb1ef2 P ad8e09f9bd4641df83c9aba7c379d4ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5399ce3a7ce41edb1b6a9223be44697 in term 1.
I20260929 20:17:47.353538 6349 leader_election.cc:302] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5399ce3a7ce41edb1b6a9223be44697, ad8e09f9bd4641df83c9aba7c379d4ed; no voters:
I20260929 20:17:47.353832 6755 raft_consensus.cc:2781] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:17:47.354275 6755 raft_consensus.cc:686] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697 [term 1 LEADER]: Becoming Leader. State: Replica: a5399ce3a7ce41edb1b6a9223be44697, State: Running, Role: LEADER
I20260929 20:17:47.354763 6755 consensus_queue.cc:237] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9504082893400f92cd853b64a29061" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40741 } } peers { permanent_uuid: "a5399ce3a7ce41edb1b6a9223be44697" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 39401 } } peers { permanent_uuid: "ad8e09f9bd4641df83c9aba7c379d4ed" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43569 } }
I20260929 20:17:47.359594 6272 catalog_manager.cc:5495] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5399ce3a7ce41edb1b6a9223be44697 (127.27.185.65). New cstate: current_term: 1 leader_uuid: "a5399ce3a7ce41edb1b6a9223be44697" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9504082893400f92cd853b64a29061" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5399ce3a7ce41edb1b6a9223be44697" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 39401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad8e09f9bd4641df83c9aba7c379d4ed" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43569 } health_report { overall_health: UNKNOWN } } }
W20260929 20:17:47.382938 6462 tablet.cc:2369] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:17:47.391094 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:47.385557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53224 (local address 127.27.185.65:39401)
0929 20:17:47.385801 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:47.385807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:47.385906 (+ 99us) server_negotiation.cc:407] Connection header received
0929 20:17:47.386175 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:47.386179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:47.386267 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:47.386433 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:17:47.386821 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.387618 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.388930 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.389340 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.389481 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:47.390000 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:17:47.390525 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:17:47.390688 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:47.390746 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 20:17:47.390826 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 20:17:47.399819 6550 raft_consensus.cc:1261] T 7fb2a464ae384e0191ac337addcb1ef2 P ad8e09f9bd4641df83c9aba7c379d4ed [term 1 FOLLOWER]: Refusing update from remote peer a5399ce3a7ce41edb1b6a9223be44697: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:17:47.399813 6684 raft_consensus.cc:1261] T 7fb2a464ae384e0191ac337addcb1ef2 P 4c9504082893400f92cd853b64a29061 [term 1 FOLLOWER]: Refusing update from remote peer a5399ce3a7ce41edb1b6a9223be44697: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:17:47.400764 6759 consensus_queue.cc:1035] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c9504082893400f92cd853b64a29061" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:17:47.401140 6755 consensus_queue.cc:1035] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad8e09f9bd4641df83c9aba7c379d4ed" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 43569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 20:17:47.407506 6596 tablet.cc:2369] T 7fb2a464ae384e0191ac337addcb1ef2 P ad8e09f9bd4641df83c9aba7c379d4ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:17:47.429191 6272 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:17:47.429507 6272 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:17:47.433449 6272 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0bbd12e490e4f23aa64b2f0620683a8: Sending DeleteTablet for 3 replicas of tablet 7fb2a464ae384e0191ac337addcb1ef2
I20260929 20:17:47.434676 6396 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fb2a464ae384e0191ac337addcb1ef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:17:47 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:53216
I20260929 20:17:47.434689 6664 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fb2a464ae384e0191ac337addcb1ef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:17:47 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:57882
I20260929 20:17:47.434801 6530 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fb2a464ae384e0191ac337addcb1ef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:17:47 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:56906
I20260929 20:17:47.437346 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 6333
I20260929 20:17:47.447984 6774 tablet_replica.cc:330] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697: stopping tablet replica
I20260929 20:17:47.448734 6774 raft_consensus.cc:2229] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:17:47.449832 6774 raft_consensus.cc:2258] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:17:47.452574 6774 ts_tablet_manager.cc:1905] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:17:47.458757 6772 tablet_replica.cc:330] T 7fb2a464ae384e0191ac337addcb1ef2 P 4c9504082893400f92cd853b64a29061: stopping tablet replica
I20260929 20:17:47.459232 6772 raft_consensus.cc:2229] T 7fb2a464ae384e0191ac337addcb1ef2 P 4c9504082893400f92cd853b64a29061 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:17:47.459828 6772 raft_consensus.cc:2258] T 7fb2a464ae384e0191ac337addcb1ef2 P 4c9504082893400f92cd853b64a29061 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:17:47.462468 6773 tablet_replica.cc:330] T 7fb2a464ae384e0191ac337addcb1ef2 P ad8e09f9bd4641df83c9aba7c379d4ed: stopping tablet replica
I20260929 20:17:47.462931 6773 raft_consensus.cc:2229] T 7fb2a464ae384e0191ac337addcb1ef2 P ad8e09f9bd4641df83c9aba7c379d4ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:17:47.463482 6773 raft_consensus.cc:2258] T 7fb2a464ae384e0191ac337addcb1ef2 P ad8e09f9bd4641df83c9aba7c379d4ed [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 29 20:17:47 dist-test-slave-4cx3 krb5kdc[6227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.65@KRBTEST.COM
I20260929 20:17:47.465742 6773 ts_tablet_manager.cc:1905] T 7fb2a464ae384e0191ac337addcb1ef2 P ad8e09f9bd4641df83c9aba7c379d4ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:17:47.466280 6772 ts_tablet_manager.cc:1905] T 7fb2a464ae384e0191ac337addcb1ef2 P 4c9504082893400f92cd853b64a29061: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:17:47.470911 6772 ts_tablet_manager.cc:1918] T 7fb2a464ae384e0191ac337addcb1ef2 P 4c9504082893400f92cd853b64a29061: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:17:47.471086 6772 log.cc:1196] T 7fb2a464ae384e0191ac337addcb1ef2 P 4c9504082893400f92cd853b64a29061: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-2/wal/wals/7fb2a464ae384e0191ac337addcb1ef2
I20260929 20:17:47.471741 6772 ts_tablet_manager.cc:1939] T 7fb2a464ae384e0191ac337addcb1ef2 P 4c9504082893400f92cd853b64a29061: Deleting consensus metadata
I20260929 20:17:47.472090 6774 ts_tablet_manager.cc:1918] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:17:47.472350 6774 log.cc:1196] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-0/wal/wals/7fb2a464ae384e0191ac337addcb1ef2
I20260929 20:17:47.472990 6774 ts_tablet_manager.cc:1939] T 7fb2a464ae384e0191ac337addcb1ef2 P a5399ce3a7ce41edb1b6a9223be44697: Deleting consensus metadata
I20260929 20:17:47.473225 6773 ts_tablet_manager.cc:1918] T 7fb2a464ae384e0191ac337addcb1ef2 P ad8e09f9bd4641df83c9aba7c379d4ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:17:47.473425 6773 log.cc:1196] T 7fb2a464ae384e0191ac337addcb1ef2 P ad8e09f9bd4641df83c9aba7c379d4ed: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790712966917858-28389-0/minicluster-data/ts-1/wal/wals/7fb2a464ae384e0191ac337addcb1ef2
I20260929 20:17:47.473830 6256 catalog_manager.cc:4841] TS 4c9504082893400f92cd853b64a29061 (127.27.185.67:40741): tablet 7fb2a464ae384e0191ac337addcb1ef2 (table test-table [id=1f182af9dedb46e09fd1c374ba08792e]) successfully deleted
I20260929 20:17:47.474016 6773 ts_tablet_manager.cc:1939] T 7fb2a464ae384e0191ac337addcb1ef2 P ad8e09f9bd4641df83c9aba7c379d4ed: Deleting consensus metadata
I20260929 20:17:47.474856 6259 catalog_manager.cc:4841] TS a5399ce3a7ce41edb1b6a9223be44697 (127.27.185.65:39401): tablet 7fb2a464ae384e0191ac337addcb1ef2 (table test-table [id=1f182af9dedb46e09fd1c374ba08792e]) successfully deleted
I20260929 20:17:47.475626 6256 catalog_manager.cc:4841] TS ad8e09f9bd4641df83c9aba7c379d4ed (127.27.185.66:43569): tablet 7fb2a464ae384e0191ac337addcb1ef2 (table test-table [id=1f182af9dedb46e09fd1c374ba08792e]) successfully deleted
I20260929 20:17:47.479900 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:47.438876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53232 (local address 127.27.185.65:39401)
0929 20:17:47.439096 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:47.439102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:47.452491 (+ 13389us) server_negotiation.cc:407] Connection header received
0929 20:17:47.453223 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:47.453228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:47.453323 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:47.453401 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:47.455182 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.456329 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.460233 (+ 3904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.460964 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.461092 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:47.472432 (+ 11340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:47.472448 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:47.472466 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:47.472525 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:47.476903 (+ 4378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:47.477676 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:47.477685 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:47.477689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:47.477873 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:47.478260 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:47.478266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:47.478269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:47.478733 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:47.479050 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:47.479207 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 20:17:47.479507 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 20:17:47.480620 6456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:47.583024 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 6333
I20260929 20:17:47.602137 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 6467
Sep 29 20:17:47 dist-test-slave-4cx3 krb5kdc[6227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.66@KRBTEST.COM
I20260929 20:17:47.616326 6746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:47.602943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56908 (local address 127.27.185.66:43569)
0929 20:17:47.603116 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:47.603120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:47.603190 (+ 70us) server_negotiation.cc:407] Connection header received
0929 20:17:47.603334 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:47.603337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:47.603467 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:47.603554 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:47.604678 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.605413 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.606564 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.606956 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.607051 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:47.611035 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:47.611049 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:47.611064 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:47.611101 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:47.613936 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:47.614543 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:47.614550 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:47.614552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:47.614667 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:47.614986 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:47.614989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:47.614991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:47.615350 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:47.615639 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:47.615876 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 20:17:47.616034 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:17:47.616984 6590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:47.705574 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 6467
I20260929 20:17:47.722553 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 6601
Sep 29 20:17:47 dist-test-slave-4cx3 krb5kdc[6227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.67@KRBTEST.COM
I20260929 20:17:47.738759 6744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:17:47.723426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57896 (local address 127.27.185.67:40741)
0929 20:17:47.723613 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 20:17:47.723619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:17:47.723639 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:17:47.723834 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:17:47.723838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:17:47.723923 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:17:47.724033 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:17:47.725323 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.726051 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.727354 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:17:47.727942 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:17:47.728086 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:17:47.732926 (+ 4840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:17:47.732944 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:17:47.732960 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:17:47.733003 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:17:47.736171 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:47.736950 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:47.736958 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:47.736961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:47.737089 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:17:47.737485 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:17:47.737488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:17:47.737490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:17:47.737835 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:17:47.738059 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 20:17:47.738316 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 20:17:47.738503 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:17:47.739524 6724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:47.830250 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 6601
I20260929 20:17:47.847982 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 6240
I20260929 20:17:47.849207 6302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:17:47.961308 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 6240
2026-09-29T20:17:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4281 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:17:48 dist-test-slave-4cx3 krb5kdc[6782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:17:48 dist-test-slave-4cx3 krb5kdc[6782](info): set up 2 sockets
Sep 29 20:17:48 dist-test-slave-4cx3 krb5kdc[6782](info): commencing operation
krb5kdc: starting...
W20260929 20:17:49.072072 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:17:49 dist-test-slave-4cx3 krb5kdc[6782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:17:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:17:49Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.185.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.185.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.185.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/rangeradmin_127.27.185.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.185.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/rangeradmin_127.27.185.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.185.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.185.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.185.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/rangerlookup_127.27.185.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.185.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/rangerlookup_127.27.185.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.185.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.185.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.185.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/HTTP_127.27.185.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.185.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/HTTP_127.27.185.84@KRBTEST.COM.keytab.
I20260929 20:17:49.402289 28389 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskPFKKvj/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/postgres -l logfile start
2026-09-29 20:17:52.079 UTC [6812] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 20:17:52.079 UTC [6812] LOG: listening on IPv4 address "127.27.185.84", port 60607
2026-09-29 20:17:52.083 UTC [6812] LOG: listening on Unix socket "/tmp/.s.PGSQL.60607"
2026-09-29 20:17:52.106 UTC [6815] LOG: database system was shut down at 2026-09-29 20:17:50 UTC
2026-09-29 20:17:52.110 UTC [6812] LOG: database system is ready to accept connections
I20260929 20:17:53.095096 28389 mini_postgres.cc:96] Postgres bound to 60607
2026-09-29 20:17:53.112 UTC [6823] FATAL: database "slave" does not exist
127.27.185.84:60607 - accepting connections
I20260929 20:17:53.113447 28389 mini_ranger.cc:161] Starting Ranger...
I20260929 20:17:53.139106 28389 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 20:17:53.679828 28389 mini_ranger.cc:87] Created ranger Postgres database
I20260929 20:17:53.679988 28389 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ranger-admin
2026-09-29 20:17:53,991 [I] DB FLAVOR :POSTGRES
2026-09-29 20:17:53,991 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 20:17:53,991 [I] Checking connection..
2026-09-29 20:17:53,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:60607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 20:17:54,263 [I] Checking connection passed.
2026-09-29 20:17:54,263 [I] --------- Verifying version history table ---------
2026-09-29 20:17:54,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:60607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 20:17:54,521 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 20:17:54,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:60607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 20:17:54,772 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 20:17:54,773 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 20:17:54,773 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:60607/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 20:17:55.020 UTC [6887] WARNING: there is no transaction in progress
2026-09-29 20:17:55,027 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 20:17:55,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:60607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 20:17:55,271 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 20:17:55,271 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 20:17:55,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:60607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 20:17:55,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:60607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 20:17:55,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:60607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-4cx3.c.gcp-upstream.internal','N') ;"
2026-09-29 20:17:55,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:60607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 20:17:56,211 [I] Table x_portal_user does not exist in database ranger
2026-09-29 20:17:56,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:60607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 20:17:56,449 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 20:17:56,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:60607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 20:17:56,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:60607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 20:17:56,953 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 20:17:56,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:60607/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 20:17:57.964 UTC [6995] WARNING: there is no transaction in progress
2026-09-29 20:17:57.981 UTC [6995] WARNING: there is no transaction in progress
2026-09-29 20:17:57.998 UTC [6995] WARNING: there is no transaction in progress
2026-09-29 20:17:58.014 UTC [6995] WARNING: there is no transaction in progress
2026-09-29 20:17:58.032 UTC [6995] WARNING: there is no transaction in progress
2026-09-29 20:17:58.052 UTC [6995] WARNING: there is no transaction in progress
2026-09-29 20:17:58.117 UTC [6995] WARNING: there is no transaction in progress
2026-09-29 20:17:58.123 UTC [6995] WARNING: there is no transaction in progress
2026-09-29 20:17:58.138 UTC [6995] WARNING: there is no transaction in progress
2026-09-29 20:17:58.149 UTC [6995] WARNING: there is no transaction in progress
2026-09-29 20:17:58.159 UTC [6995] WARNING: there is no transaction in progress
2026-09-29 20:17:58.173 UTC [6995] WARNING: there is no transaction in progress
2026-09-29 20:17:58.183 UTC [6995] WARNING: there is no transaction in progress
2026-09-29 20:17:58.190 UTC [6995] WARNING: there is no transaction in progress
2026-09-29 20:17:58.197 UTC [6995] WARNING: there is no transaction in progress
2026-09-29 20:17:58.204 UTC [6995] WARNING: there is no transaction in progress
2026-09-29 20:17:58,856 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 20:17:58,856 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:60607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 20:17:59,115 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 20:17:59,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:60607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 20:17:59,366 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 20:17:59,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:60607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 20:17:59,614 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 20:17:59,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:60607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 20:17:59,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:60607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 20:18:00,107 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:60607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-4cx3.c.gcp-upstream.internal';"
2026-09-29 20:18:00,338 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 20:18:00,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:60607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 20:18:00,596 [I] DB_PATCHES have already been applied
I20260929 20:18:00.602123 28389 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ranger-admin:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskPFKKvj/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 20:18:00.602234 28389 mini_ranger.cc:193] Using host: 127.27.185.84
I20260929 20:18:00.616961 28389 mini_ranger.cc:227] Ranger admin URL: http://127.27.185.84:60483
Sep 29, 2026 8:18:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 8:18:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 8:18:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 8:18:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 8:18:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 8:18:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 8:18:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 20:18:01.823355 28389 mini_ranger.cc:160] Time spent starting Ranger: real 8.710s user 0.001s sys 0.038s
Sep 29, 2026 8:18:20 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 8:18:20 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 29, 2026 8:18:20 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 8:18:21 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 8:18:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 8:18:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 8:18:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 8:18:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 8:18:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 8:18:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 8:18:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 8:18:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 8:18:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 8:18:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 8:18:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 8:18:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 8:18:21 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 8:18:21 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 20:18:22.555045 28389 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.185.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.185.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.185.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/rangerkms_127.27.185.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.185.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/rangerkms_127.27.185.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.185.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.185.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.185.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/HTTP_127.27.185.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.185.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/HTTP_127.27.185.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 20:18:22 dist-test-slave-4cx3 krb5kdc[6782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713102, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 20:18:22.775795 28389 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 20:18:22.811065 28389 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 20:18:23.740330 28389 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 20:18:23.740474 28389 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ranger-kms
I20260929 20:18:23.740645 28389 mini_ranger_kms.cc:227] Using postgres at 127.27.185.84:60607
2026-09-29 20:18:24,168 [I] DB FLAVOR :POSTGRES
2026-09-29 20:18:24,168 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 20:18:24,168 [I] Checking connection
2026-09-29 20:18:24,169 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:60607/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 20:18:24,435 [I] connection success
2026-09-29 20:18:24,435 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 20:18:24,435 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 20:18:24,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:60607/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 20:18:24,707 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 20:18:24,707 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 20:18:24,707 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 20:18:24,708 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:60607/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 20:18:24,998 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 20:18:25.468997 28389 mini_ranger_kms.cc:310] Created kms service
I20260929 20:18:25.667075 28389 mini_ranger_kms.cc:326] Created kudu user
I20260929 20:18:25.713331 28389 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 20:18:25 dist-test-slave-4cx3 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713102, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.185.84@KRBTEST.COM
I20260929 20:18:26.119045 28389 mini_ranger_kms.cc:384] Added ranger policy
I20260929 20:18:26.119452 28389 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ranger-kms:/tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 20:18:26.119535 28389 mini_ranger_kms.cc:240] Using host: 127.27.185.84
I20260929 20:18:26.137661 28389 mini_ranger_kms.cc:276] Ranger KMS PID: 7175
I20260929 20:18:26.137908 28389 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.185.84:46973
Sep 29, 2026 8:18:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 8:18:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 8:18:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 8:18:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 8:18:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 8:18:27 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 20:18:28.441287 28389 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.666s user 0.120s sys 0.331s
I20260929 20:18:28.441550 28389 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 20:18:28.441771 28389 mini_ranger_kms.cc:401] 127.27.185.84:46973/kms/v1/keys
Sep 29 20:18:28 dist-test-slave-4cx3 krb5kdc[6782](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790713108, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.185.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 20:18:29 dist-test-slave-4cx3 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713108, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.185.84@KRBTEST.COM for HTTP/127.27.185.84@KRBTEST.COM
Sep 29 20:18:29 dist-test-slave-4cx3 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713108, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.185.84@KRBTEST.COM for HTTP/127.27.185.84@KRBTEST.COM
Sep 29 20:18:31 dist-test-slave-4cx3 krb5kdc[6782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
I20260929 20:18:31.724846 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:35331
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:36731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:35331
--ranger_config_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.185.84:46973/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:18:32.074306 7226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:18:32.074615 7226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:18:32.074767 7226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:18:32.082369 7226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:18:32.082502 7226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:18:32.082588 7226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:18:32.082701 7226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:18:32.082768 7226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:18:32.093521 7226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:36731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.185.84:46973/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:35331
--ranger_config_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:35331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:18:32.094898 7226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:18:32.096346 7226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:18:32.109544 7232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:18:32.109755 7231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:18:32.109733 7234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:18:32.110057 7226 server_base.cc:1034] running on GCE node
I20260929 20:18:32.111188 7226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:18:32.112758 7226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:18:32.114042 7226 hybrid_clock.cc:648] HybridClock initialized: now 1790713112113989 us; error 52 us; skew 500 ppm
I20260929 20:18:32.114423 7226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:18:32 dist-test-slave-4cx3 krb5kdc[6782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713112, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:18:32.121102 7226 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:18:32.122901 7226 webserver.cc:460] Webserver started at http://127.27.185.126:42033/ using document root <none> and password file <none>
I20260929 20:18:32.123683 7226 fs_manager.cc:359] Metadata directory not provided
I20260929 20:18:32.123782 7226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:18:32.124145 7226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:18:32 dist-test-slave-4cx3 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713112, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for HTTP/127.27.185.84@KRBTEST.COM
I20260929 20:18:32.174587 7226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "861317cf78fb4e88b4242f1c7f1ba054"
format_stamp: "Formatted at 2026-09-29 20:18:32 on dist-test-slave-4cx3"
server_key: "5856cd14760928ae6d9ab15424bdd5db"
server_key_iv: "f54882c76eb21c5fef2772ca64eed290"
server_key_version: "kuduclusterkey@0"
I20260929 20:18:32.175506 7226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "861317cf78fb4e88b4242f1c7f1ba054"
format_stamp: "Formatted at 2026-09-29 20:18:32 on dist-test-slave-4cx3"
server_key: "5856cd14760928ae6d9ab15424bdd5db"
server_key_iv: "f54882c76eb21c5fef2772ca64eed290"
server_key_version: "kuduclusterkey@0"
I20260929 20:18:32.181143 7226 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 20:18:32.202733 7241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:18:32.204495 7226 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 20:18:32.204699 7226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "861317cf78fb4e88b4242f1c7f1ba054"
format_stamp: "Formatted at 2026-09-29 20:18:32 on dist-test-slave-4cx3"
server_key: "5856cd14760928ae6d9ab15424bdd5db"
server_key_iv: "f54882c76eb21c5fef2772ca64eed290"
server_key_version: "kuduclusterkey@0"
I20260929 20:18:32.204967 7226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:18:32.218906 7226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:18:32.223325 7226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:18:32.223685 7226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:18:32.244796 7226 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:35331
I20260929 20:18:32.244814 7292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:35331 every 8 connection(s)
I20260929 20:18:32.246181 7226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:18:32.249992 7293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:18:32.255329 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 7226
I20260929 20:18:32.255565 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:18:32.256356 28389 external_mini_cluster.cc:1428] Setting key 727ce73e5c23028447b09b7e0e97fff1
I20260929 20:18:32.262879 7293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 861317cf78fb4e88b4242f1c7f1ba054: Bootstrap starting.
I20260929 20:18:32.268077 7293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 861317cf78fb4e88b4242f1c7f1ba054: Neither blocks nor log segments found. Creating new log.
I20260929 20:18:32.269795 7293 log.cc:824] T 00000000000000000000000000000000 P 861317cf78fb4e88b4242f1c7f1ba054: Log is configured to *not* fsync() on all Append() calls
I20260929 20:18:32.273001 7293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 861317cf78fb4e88b4242f1c7f1ba054: No bootstrap required, opened a new log
I20260929 20:18:32.282454 7293 raft_consensus.cc:348] T 00000000000000000000000000000000 P 861317cf78fb4e88b4242f1c7f1ba054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861317cf78fb4e88b4242f1c7f1ba054" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 35331 } }
I20260929 20:18:32.282977 7293 raft_consensus.cc:374] T 00000000000000000000000000000000 P 861317cf78fb4e88b4242f1c7f1ba054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:18:32.283161 7293 raft_consensus.cc:729] T 00000000000000000000000000000000 P 861317cf78fb4e88b4242f1c7f1ba054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861317cf78fb4e88b4242f1c7f1ba054, State: Initialized, Role: FOLLOWER
I20260929 20:18:32.283926 7293 consensus_queue.cc:260] T 00000000000000000000000000000000 P 861317cf78fb4e88b4242f1c7f1ba054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861317cf78fb4e88b4242f1c7f1ba054" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 35331 } }
I20260929 20:18:32.284175 7293 raft_consensus.cc:388] T 00000000000000000000000000000000 P 861317cf78fb4e88b4242f1c7f1ba054 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Sep 29 20:18:32 dist-test-slave-4cx3 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:18:32.284343 7293 raft_consensus.cc:482] T 00000000000000000000000000000000 P 861317cf78fb4e88b4242f1c7f1ba054 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:18:32.284528 7293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 861317cf78fb4e88b4242f1c7f1ba054 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:18:32.287708 7293 raft_consensus.cc:504] T 00000000000000000000000000000000 P 861317cf78fb4e88b4242f1c7f1ba054 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861317cf78fb4e88b4242f1c7f1ba054" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 35331 } }
I20260929 20:18:32.288352 7293 leader_election.cc:302] T 00000000000000000000000000000000 P 861317cf78fb4e88b4242f1c7f1ba054 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 861317cf78fb4e88b4242f1c7f1ba054; no voters:
I20260929 20:18:32.289160 7293 leader_election.cc:288] T 00000000000000000000000000000000 P 861317cf78fb4e88b4242f1c7f1ba054 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:18:32.289314 7298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 861317cf78fb4e88b4242f1c7f1ba054 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:18:32.297207 7296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:18:32.258599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36600 (local address 127.27.185.126:35331)
0929 20:18:32.269708 (+ 11109us) server_negotiation.cc:206] Beginning negotiation
0929 20:18:32.269726 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 20:18:32.270159 (+ 433us) server_negotiation.cc:407] Connection header received
0929 20:18:32.272720 (+ 2561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:18:32.272751 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:18:32.273267 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:18:32.273717 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:18:32.275315 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:18:32.277350 (+ 2035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:18:32.279223 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:18:32.280395 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:18:32.280595 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:18:32.285617 (+ 5022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:18:32.285633 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:18:32.285655 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:18:32.285713 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:18:32.290043 (+ 4330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:18:32.291267 (+ 1224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:18:32.291280 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:18:32.291293 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:18:32.291562 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:18:32.292192 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:18:32.292200 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:18:32.292206 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:18:32.293175 (+ 969us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:18:32.294830 (+ 1655us) server_negotiation.cc:1035] Waiting for connection context
0929 20:18:32.295207 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 20:18:32.295803 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10834,"thread_start_us":10586,"threads_started":1}
I20260929 20:18:32.297912 7293 sys_catalog.cc:564] T 00000000000000000000000000000000 P 861317cf78fb4e88b4242f1c7f1ba054 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:18:32.299170 7298 raft_consensus.cc:686] T 00000000000000000000000000000000 P 861317cf78fb4e88b4242f1c7f1ba054 [term 1 LEADER]: Becoming Leader. State: Replica: 861317cf78fb4e88b4242f1c7f1ba054, State: Running, Role: LEADER
I20260929 20:18:32.299872 7298 consensus_queue.cc:237] T 00000000000000000000000000000000 P 861317cf78fb4e88b4242f1c7f1ba054 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861317cf78fb4e88b4242f1c7f1ba054" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 35331 } }
I20260929 20:18:32.306593 7293 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 20:18:32.306700 7300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 861317cf78fb4e88b4242f1c7f1ba054 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 861317cf78fb4e88b4242f1c7f1ba054. Latest consensus state: current_term: 1 leader_uuid: "861317cf78fb4e88b4242f1c7f1ba054" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861317cf78fb4e88b4242f1c7f1ba054" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 35331 } } }
I20260929 20:18:32.307093 7300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 861317cf78fb4e88b4242f1c7f1ba054 [sys.catalog]: This master's current role is: LEADER
I20260929 20:18:32.307643 7299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 861317cf78fb4e88b4242f1c7f1ba054 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "861317cf78fb4e88b4242f1c7f1ba054" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861317cf78fb4e88b4242f1c7f1ba054" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 35331 } } }
I20260929 20:18:32.307909 7299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 861317cf78fb4e88b4242f1c7f1ba054 [sys.catalog]: This master's current role is: LEADER
I20260929 20:18:32.316612 7312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:18:32.323853 7312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:18:32.334730 7312 catalog_manager.cc:1347] Generated new cluster ID: 5b0b1d56f4514e09ba073b5923dbca83
I20260929 20:18:32.334929 7312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:18:32.345361 7312 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:18:32.347340 7312 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:18:32.360821 7312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 861317cf78fb4e88b4242f1c7f1ba054: Generated new TSK 0
I20260929 20:18:32.362375 7312 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
I20260929 20:18:33.257676 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.185.84:46973/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:0
--local_ip_for_outbound_sockets=127.27.185.65
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:35331
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:36731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:18:33 dist-test-slave-4cx3 krb5kdc[6782](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790713113, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 20:18:33.700486 7338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:18:33.701014 7338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:18:33.701223 7338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:18:33.713439 7338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:18:33.713662 7338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:18:33.713888 7338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:18:33.730022 7338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:36731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.185.84:46973/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:35331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:18:33.731443 7338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:18:33.732841 7338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:18:33.748118 7349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:18:33.748154 7351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:18:33.748214 7348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:18:33.748862 7338 server_base.cc:1034] running on GCE node
I20260929 20:18:33.749563 7338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:18:33.750670 7338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:18:33.751889 7338 hybrid_clock.cc:648] HybridClock initialized: now 1790713113751837 us; error 47 us; skew 500 ppm
I20260929 20:18:33.752202 7338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:18:33 dist-test-slave-4cx3 krb5kdc[6782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713113, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:18:33.756044 7338 init.cc:377] Logged in from keytab as kudu/127.27.185.65@KRBTEST.COM (short username kudu)
I20260929 20:18:33.758314 7338 webserver.cc:460] Webserver started at http://127.27.185.65:45449/ using document root <none> and password file <none>
I20260929 20:18:33.759123 7338 fs_manager.cc:359] Metadata directory not provided
I20260929 20:18:33.759291 7338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:18:33.759717 7338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:18:33 dist-test-slave-4cx3 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713113, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for HTTP/127.27.185.84@KRBTEST.COM
I20260929 20:18:33.782656 7338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-0/data/instance:
uuid: "214e50676fc5411e8f1629f1728c0848"
format_stamp: "Formatted at 2026-09-29 20:18:33 on dist-test-slave-4cx3"
server_key: "69c7f66f7de683a58d683ef48deefe8f"
server_key_iv: "73c95a8d61084f9b99dd4a52d11b1b28"
server_key_version: "kuduclusterkey@0"
I20260929 20:18:33.784003 7338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance:
uuid: "214e50676fc5411e8f1629f1728c0848"
format_stamp: "Formatted at 2026-09-29 20:18:33 on dist-test-slave-4cx3"
server_key: "69c7f66f7de683a58d683ef48deefe8f"
server_key_iv: "73c95a8d61084f9b99dd4a52d11b1b28"
server_key_version: "kuduclusterkey@0"
I20260929 20:18:33.792250 7338 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260929 20:18:33.832026 7358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:18:33.840106 7338 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.009s sys 0.004s
I20260929 20:18:33.840440 7338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "214e50676fc5411e8f1629f1728c0848"
format_stamp: "Formatted at 2026-09-29 20:18:33 on dist-test-slave-4cx3"
server_key: "69c7f66f7de683a58d683ef48deefe8f"
server_key_iv: "73c95a8d61084f9b99dd4a52d11b1b28"
server_key_version: "kuduclusterkey@0"
I20260929 20:18:33.840803 7338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:18:33.871202 7338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:18:33.876407 7338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:18:33.876729 7338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:18:33.878119 7338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:18:33.879652 7338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:18:33.879756 7338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:18:33.879900 7338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:18:33.879981 7338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:18:33 dist-test-slave-4cx3 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713113, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for HTTP/127.27.185.84@KRBTEST.COM
I20260929 20:18:34.272861 7470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:41391 every 8 connection(s)
I20260929 20:18:34.273226 7338 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:41391
I20260929 20:18:34.274907 7338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:18:34.279496 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 7338
I20260929 20:18:34.279708 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance
I20260929 20:18:34.280969 28389 external_mini_cluster.cc:1428] Setting key 43eddc4557cca98fa74214dea7c4d4a5
WARNING: no policy specified for kudu/127.27.185.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.66@KRBTEST.COM" created.
Sep 29 20:18:34 dist-test-slave-4cx3 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713113, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:18:34.344262 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:18:34.298231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:37513 (local address 127.27.185.126:35331)
0929 20:18:34.302852 (+ 4621us) server_negotiation.cc:206] Beginning negotiation
0929 20:18:34.302858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:18:34.306324 (+ 3466us) server_negotiation.cc:407] Connection header received
0929 20:18:34.312532 (+ 6208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:18:34.312539 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:18:34.312646 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:18:34.312727 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:18:34.315017 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:18:34.316140 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:18:34.318428 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:18:34.319010 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:18:34.319104 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:18:34.334983 (+ 15879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:18:34.334992 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:18:34.334996 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:18:34.335053 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:18:34.338584 (+ 3531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:18:34.339755 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:18:34.339761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:18:34.339765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:18:34.339909 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:18:34.340558 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:18:34.340564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:18:34.340567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:18:34.340950 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:18:34.341124 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 20:18:34.343617 (+ 2493us) server_negotiation.cc:299] Negotiation successful
0929 20:18:34.343868 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4465,"thread_start_us":4282,"threads_started":1}
I20260929 20:18:34.346832 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:18:34.297581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:35331 (local address 127.27.185.65:37513)
0929 20:18:34.306135 (+ 8554us) negotiation.cc:107] Waiting for socket to connect
0929 20:18:34.306196 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 20:18:34.308687 (+ 2491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:18:34.313330 (+ 4643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:18:34.313343 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:18:34.313980 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:18:34.314812 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:18:34.314837 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:18:34.316334 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:18:34.316338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:18:34.318258 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:18:34.318272 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:18:34.319758 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:18:34.319769 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:18:34.320058 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:18:34.320850 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:18:34.320879 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:18:34.334624 (+ 13745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:18:34.338825 (+ 4201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:18:34.338835 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:18:34.338855 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:18:34.339597 (+ 742us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:18:34.340080 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:18:34.340084 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:18:34.340087 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:18:34.340416 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:18:34.341230 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:18:34.341239 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:18:34.341714 (+ 475us) client_negotiation.cc:769] Sending connection context
0929 20:18:34.344804 (+ 3090us) client_negotiation.cc:240] Negotiation successful
0929 20:18:34.345375 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8209,"thread_start_us":7967,"threads_started":1}
I20260929 20:18:34.351063 7471 heartbeater.cc:347] Connected to a master server at 127.27.185.126:35331
I20260929 20:18:34.351500 7471 heartbeater.cc:464] Registering TS with master...
I20260929 20:18:34.352391 7471 heartbeater.cc:511] Master 127.27.185.126:35331 requested a full tablet report, sending...
I20260929 20:18:34.355124 7258 ts_manager.cc:194] Registered new tserver with Master: 214e50676fc5411e8f1629f1728c0848 (127.27.185.65:41391)
Sep 29 20:18:34 dist-test-slave-4cx3 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713113, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for HTTP/127.27.185.84@KRBTEST.COM
I20260929 20:18:34.358697 7258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.65@KRBTEST.COM'} at 127.27.185.65:37513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
W20260929 20:18:34.395150 28389 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.27.185.66: real 0.111s user 0.005s sys 0.012s
I20260929 20:18:34.396203 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.185.84:46973/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:0
--local_ip_for_outbound_sockets=127.27.185.66
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:35331
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:36731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:18:34.851356 7478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:18:34.851709 7478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:18:34.851801 7478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:18:34.859051 7478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:18:34.859162 7478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:18:34.859272 7478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:18:34.869879 7478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:36731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.185.84:46973/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:35331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:18:34.871202 7478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:18:34.872516 7478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:18:34.888379 7491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:18:34.888355 7492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:18:34.889344 7494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:18:34.889854 7478 server_base.cc:1034] running on GCE node
I20260929 20:18:34.890491 7478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:18:34.891541 7478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:18:34.892776 7478 hybrid_clock.cc:648] HybridClock initialized: now 1790713114892715 us; error 53 us; skew 500 ppm
I20260929 20:18:34.893121 7478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:18:34 dist-test-slave-4cx3 krb5kdc[6782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713114, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:18:34.896266 7478 init.cc:377] Logged in from keytab as kudu/127.27.185.66@KRBTEST.COM (short username kudu)
I20260929 20:18:34.897871 7478 webserver.cc:460] Webserver started at http://127.27.185.66:46357/ using document root <none> and password file <none>
I20260929 20:18:34.898586 7478 fs_manager.cc:359] Metadata directory not provided
I20260929 20:18:34.898711 7478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:18:34.899089 7478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:18:34 dist-test-slave-4cx3 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713114, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for HTTP/127.27.185.84@KRBTEST.COM
I20260929 20:18:34.915674 7478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-1/data/instance:
uuid: "32f91d348a1a4c37bd9c0d703394b172"
format_stamp: "Formatted at 2026-09-29 20:18:34 on dist-test-slave-4cx3"
server_key: "251148aed8648bc4b5393d9b3078e60e"
server_key_iv: "203c2a3f3bc31173cdb3d3be2ed7fef4"
server_key_version: "kuduclusterkey@0"
I20260929 20:18:34.916561 7478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance:
uuid: "32f91d348a1a4c37bd9c0d703394b172"
format_stamp: "Formatted at 2026-09-29 20:18:34 on dist-test-slave-4cx3"
server_key: "251148aed8648bc4b5393d9b3078e60e"
server_key_iv: "203c2a3f3bc31173cdb3d3be2ed7fef4"
server_key_version: "kuduclusterkey@0"
I20260929 20:18:34.922115 7478 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:18:34.938128 7501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:18:34.939815 7478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:18:34.940021 7478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "32f91d348a1a4c37bd9c0d703394b172"
format_stamp: "Formatted at 2026-09-29 20:18:34 on dist-test-slave-4cx3"
server_key: "251148aed8648bc4b5393d9b3078e60e"
server_key_iv: "203c2a3f3bc31173cdb3d3be2ed7fef4"
server_key_version: "kuduclusterkey@0"
I20260929 20:18:34.940253 7478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:18:34.966841 7478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:18:34.971750 7478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:18:34.972072 7478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:18:34.973098 7478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:18:34.974571 7478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:18:34.974674 7478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:18:34.974829 7478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:18:34.974901 7478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:18:35.017632 7478 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:38437
I20260929 20:18:35.017657 7613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:38437 every 8 connection(s)
I20260929 20:18:35.019097 7478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:18:35.021010 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 7478
I20260929 20:18:35.021190 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance
I20260929 20:18:35.021732 28389 external_mini_cluster.cc:1428] Setting key 0f3b6284f24ea1ee9f1317b11a52cc24
Sep 29 20:18:35 dist-test-slave-4cx3 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713114, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:18:35.048262 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:18:35.022376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:35067 (local address 127.27.185.126:35331)
0929 20:18:35.022926 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0929 20:18:35.022933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:18:35.027445 (+ 4512us) server_negotiation.cc:407] Connection header received
0929 20:18:35.028538 (+ 1093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:18:35.028545 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:18:35.028638 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:18:35.028776 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:18:35.031149 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.032431 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.034964 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.035581 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.035752 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:18:35.039844 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:18:35.039851 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:18:35.039855 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:18:35.039901 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:18:35.044070 (+ 4169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:18:35.045447 (+ 1377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:18:35.045451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:18:35.045454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:18:35.045602 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:18:35.046129 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:18:35.046136 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:18:35.046140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:18:35.046488 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:18:35.046664 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 20:18:35.047534 (+ 870us) server_negotiation.cc:299] Negotiation successful
0929 20:18:35.047800 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":276,"threads_started":1}
I20260929 20:18:35.049839 7617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:18:35.023659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:35331 (local address 127.27.185.66:35067)
0929 20:18:35.025074 (+ 1415us) negotiation.cc:107] Waiting for socket to connect
0929 20:18:35.025134 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 20:18:35.028152 (+ 3018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:18:35.029121 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:18:35.029139 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:18:35.029884 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:18:35.030903 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:18:35.030938 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.032652 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.032658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:18:35.034769 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:18:35.034796 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.035871 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.035881 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:18:35.036181 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:18:35.037000 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:18:35.037032 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:18:35.039526 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:18:35.044357 (+ 4831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:18:35.044379 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:18:35.044402 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:18:35.045225 (+ 823us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:18:35.045754 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:18:35.045759 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:18:35.045764 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:18:35.045984 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:18:35.046654 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:18:35.046662 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:18:35.047071 (+ 409us) client_negotiation.cc:769] Sending connection context
0929 20:18:35.047400 (+ 329us) client_negotiation.cc:240] Negotiation successful
0929 20:18:35.048036 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1080,"thread_start_us":840,"threads_started":1}
I20260929 20:18:35.051983 7614 heartbeater.cc:347] Connected to a master server at 127.27.185.126:35331
I20260929 20:18:35.052366 7614 heartbeater.cc:464] Registering TS with master...
I20260929 20:18:35.053210 7614 heartbeater.cc:511] Master 127.27.185.126:35331 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.185.67@KRBTEST.COM; defaulting to no policy
I20260929 20:18:35.055023 7258 ts_manager.cc:194] Registered new tserver with Master: 32f91d348a1a4c37bd9c0d703394b172 (127.27.185.66:38437)
I20260929 20:18:35.056095 7258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.66@KRBTEST.COM'} at 127.27.185.66:35067
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
I20260929 20:18:35.095142 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.185.84:46973/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:0
--local_ip_for_outbound_sockets=127.27.185.67
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:35331
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:36731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:18:35.362984 7471 heartbeater.cc:503] Master 127.27.185.126:35331 was elected leader, sending a full tablet report...
W20260929 20:18:35.448863 7620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:18:35.449199 7620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:18:35.449337 7620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:18:35.457057 7620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:18:35.457168 7620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:18:35.457278 7620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:18:35.466988 7620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:36731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.185.84:46973/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:35331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:18:35.468263 7620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:18:35.469517 7620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:18:35.483150 7626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:18:35.483206 7625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:18:35.483217 7628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:18:35.483919 7620 server_base.cc:1034] running on GCE node
I20260929 20:18:35.484575 7620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:18:35.485544 7620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:18:35.486778 7620 hybrid_clock.cc:648] HybridClock initialized: now 1790713115486715 us; error 58 us; skew 500 ppm
I20260929 20:18:35.487108 7620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:18:35 dist-test-slave-4cx3 krb5kdc[6782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713115, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:18:35.490356 7620 init.cc:377] Logged in from keytab as kudu/127.27.185.67@KRBTEST.COM (short username kudu)
I20260929 20:18:35.491933 7620 webserver.cc:460] Webserver started at http://127.27.185.67:44389/ using document root <none> and password file <none>
I20260929 20:18:35.492617 7620 fs_manager.cc:359] Metadata directory not provided
I20260929 20:18:35.492765 7620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:18:35.493079 7620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:18:35 dist-test-slave-4cx3 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713115, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for HTTP/127.27.185.84@KRBTEST.COM
I20260929 20:18:35.510171 7620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-2/data/instance:
uuid: "6db9cecaaf4a4fc0bf1c8c2b859cf714"
format_stamp: "Formatted at 2026-09-29 20:18:35 on dist-test-slave-4cx3"
server_key: "5d8d5f5dccb6e75064d56a2bc2fd283a"
server_key_iv: "c8c605e046da89e1a94dbfa651106692"
server_key_version: "kuduclusterkey@0"
I20260929 20:18:35.511289 7620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance:
uuid: "6db9cecaaf4a4fc0bf1c8c2b859cf714"
format_stamp: "Formatted at 2026-09-29 20:18:35 on dist-test-slave-4cx3"
server_key: "5d8d5f5dccb6e75064d56a2bc2fd283a"
server_key_iv: "c8c605e046da89e1a94dbfa651106692"
server_key_version: "kuduclusterkey@0"
I20260929 20:18:35.518044 7620 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 20:18:35.534970 7635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:18:35.536585 7620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:18:35.536783 7620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "6db9cecaaf4a4fc0bf1c8c2b859cf714"
format_stamp: "Formatted at 2026-09-29 20:18:35 on dist-test-slave-4cx3"
server_key: "5d8d5f5dccb6e75064d56a2bc2fd283a"
server_key_iv: "c8c605e046da89e1a94dbfa651106692"
server_key_version: "kuduclusterkey@0"
I20260929 20:18:35.537014 7620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:18:35.554965 7620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:18:35.558894 7620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:18:35.559201 7620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:18:35.560210 7620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:18:35.561621 7620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:18:35.561718 7620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:18:35.561880 7620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:18:35.561955 7620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:18:35.599555 7620 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:41651
I20260929 20:18:35.599671 7747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:41651 every 8 connection(s)
I20260929 20:18:35.600867 7620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:18:35.602952 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 7620
I20260929 20:18:35.603145 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance
I20260929 20:18:35.603746 28389 external_mini_cluster.cc:1428] Setting key 77a77577e69ccd7a4eff4001e8d70210
Sep 29 20:18:35 dist-test-slave-4cx3 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713115, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:18:35.622100 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:18:35.603913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:47603 (local address 127.27.185.126:35331)
0929 20:18:35.604493 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0929 20:18:35.604501 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:18:35.605642 (+ 1141us) server_negotiation.cc:407] Connection header received
0929 20:18:35.607678 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:18:35.607683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:18:35.607760 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:18:35.607851 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:18:35.609514 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.610330 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.611915 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.612356 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.612449 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:18:35.615741 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:18:35.615746 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:18:35.615749 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:18:35.615792 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:18:35.618973 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:18:35.619774 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:18:35.619778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:18:35.619780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:18:35.619896 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:18:35.620252 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:18:35.620255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:18:35.620257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:18:35.620509 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:18:35.620615 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 20:18:35.621682 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0929 20:18:35.621840 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":283,"threads_started":1}
I20260929 20:18:35.623057 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:18:35.604628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:35331 (local address 127.27.185.67:47603)
0929 20:18:35.605488 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0929 20:18:35.605538 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 20:18:35.607350 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:18:35.608175 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:18:35.608183 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:18:35.608642 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:18:35.609331 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:18:35.609351 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.610469 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.610473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:18:35.611782 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:18:35.611793 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.612537 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.612541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:18:35.612746 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:18:35.613468 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:18:35.613490 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:18:35.615526 (+ 2036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:18:35.619117 (+ 3591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:18:35.619127 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:18:35.619143 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:18:35.619663 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:18:35.619991 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:18:35.619994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:18:35.619996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:18:35.620157 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:18:35.621034 (+ 877us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:18:35.621040 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:18:35.621317 (+ 277us) client_negotiation.cc:769] Sending connection context
0929 20:18:35.621572 (+ 255us) client_negotiation.cc:240] Negotiation successful
0929 20:18:35.621932 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":369,"threads_started":1}
I20260929 20:18:35.624409 7748 heartbeater.cc:347] Connected to a master server at 127.27.185.126:35331
I20260929 20:18:35.624743 7748 heartbeater.cc:464] Registering TS with master...
I20260929 20:18:35.625439 7748 heartbeater.cc:511] Master 127.27.185.126:35331 requested a full tablet report, sending...
I20260929 20:18:35.626817 7258 ts_manager.cc:194] Registered new tserver with Master: 6db9cecaaf4a4fc0bf1c8c2b859cf714 (127.27.185.67:41651)
I20260929 20:18:35.627697 7258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.67@KRBTEST.COM'} at 127.27.185.67:47603
I20260929 20:18:35.631534 28389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:18:35.648931 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:18:35.635911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35086 (local address 127.27.185.126:35331)
0929 20:18:35.636078 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 20:18:35.636083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:18:35.636639 (+ 556us) server_negotiation.cc:407] Connection header received
0929 20:18:35.636836 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:18:35.636841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:18:35.636950 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:18:35.637056 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:18:35.638447 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.639152 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.640676 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.641116 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.641255 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:18:35.643785 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:18:35.643794 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:18:35.643803 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:18:35.643837 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:18:35.646433 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:18:35.647064 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:18:35.647068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:18:35.647070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:18:35.647198 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:18:35.647701 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:18:35.647705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:18:35.647707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:18:35.647958 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:18:35.648146 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 20:18:35.648430 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 20:18:35.648612 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 20:18:35.652956 7258 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:18:35.660604 7258 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:18:35.685856 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:18:35.674419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:41651 (local address 127.0.0.1:41340)
0929 20:18:35.675332 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0929 20:18:35.675406 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0929 20:18:35.675602 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:18:35.677085 (+ 1483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:18:35.677095 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:18:35.677140 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:18:35.677914 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:18:35.677933 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.680000 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.680006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:18:35.683164 (+ 3158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:18:35.683193 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.684817 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.684823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:18:35.685040 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:18:35.685096 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 20:18:35.685255 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 20:18:35.685454 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":1,"thread_start_us":362,"threads_started":1}
I20260929 20:18:35.686614 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:18:35.674858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41340 (local address 127.27.185.67:41651)
0929 20:18:35.676328 (+ 1470us) server_negotiation.cc:206] Beginning negotiation
0929 20:18:35.676337 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:18:35.676374 (+ 37us) server_negotiation.cc:407] Connection header received
0929 20:18:35.676489 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:18:35.676496 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:18:35.676807 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:18:35.677391 (+ 584us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:18:35.678194 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.679825 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.683347 (+ 3522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.684600 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.684753 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:18:35.685004 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 20:18:35.685299 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 20:18:35.685460 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1290,"thread_start_us":1099,"threads_started":1}
I20260929 20:18:35.686789 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:18:35.676323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:38437 (local address 127.0.0.1:47040)
0929 20:18:35.678929 (+ 2606us) negotiation.cc:107] Waiting for socket to connect
0929 20:18:35.678949 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:18:35.679072 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:18:35.679823 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:18:35.679828 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:18:35.679855 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:18:35.680340 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:18:35.680352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.681800 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.681805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:18:35.683867 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:18:35.683886 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.685966 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.685974 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:18:35.686143 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:18:35.686181 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 20:18:35.686270 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 20:18:35.686407 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2479,"mutex_wait_us":23,"thread_start_us":2717,"threads_started":1}
I20260929 20:18:35.687350 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:18:35.676583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47040 (local address 127.27.185.66:38437)
0929 20:18:35.679300 (+ 2717us) server_negotiation.cc:206] Beginning negotiation
0929 20:18:35.679308 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:18:35.679336 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:18:35.679458 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:18:35.679465 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:18:35.679633 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:18:35.679823 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:18:35.680501 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.681640 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.684050 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.685511 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.685638 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:18:35.685821 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 20:18:35.686922 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0929 20:18:35.687086 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2101,"thread_start_us":1916,"threads_started":1}
I20260929 20:18:35.688360 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:18:35.674452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:41391 (local address 127.0.0.1:58212)
0929 20:18:35.678891 (+ 4439us) negotiation.cc:107] Waiting for socket to connect
0929 20:18:35.678916 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 20:18:35.679122 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:18:35.680524 (+ 1402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:18:35.680527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:18:35.680550 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:18:35.681473 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:18:35.681487 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.683411 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.683417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:18:35.685343 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:18:35.685358 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.687567 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.687573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:18:35.687740 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:18:35.687773 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:18:35.687876 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 20:18:35.688008 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4314,"mutex_wait_us":1,"thread_start_us":2795,"threads_started":1}
I20260929 20:18:35.689240 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:18:35.674543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58212 (local address 127.27.185.65:41391)
0929 20:18:35.678093 (+ 3550us) server_negotiation.cc:206] Beginning negotiation
0929 20:18:35.678101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:18:35.679018 (+ 917us) server_negotiation.cc:407] Connection header received
0929 20:18:35.679328 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:18:35.679336 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:18:35.679615 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:18:35.679823 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:18:35.681705 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.683262 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.685527 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.686998 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.687111 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:18:35.687332 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 20:18:35.688723 (+ 1391us) server_negotiation.cc:299] Negotiation successful
0929 20:18:35.688903 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3351,"thread_start_us":4297,"threads_started":1}
I20260929 20:18:35.690922 7683 tablet_service.cc:1467] Processing CreateTablet for tablet 492a05dcabee4d7fa8abde9aabb6e7bc (DEFAULT_TABLE table=test-table [id=9a8d96fd91834f199bf0fb719f92954d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:18:35.690959 7549 tablet_service.cc:1467] Processing CreateTablet for tablet 492a05dcabee4d7fa8abde9aabb6e7bc (DEFAULT_TABLE table=test-table [id=9a8d96fd91834f199bf0fb719f92954d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:18:35.692529 7683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 492a05dcabee4d7fa8abde9aabb6e7bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:18:35.692821 7549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 492a05dcabee4d7fa8abde9aabb6e7bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:18:35.693292 7406 tablet_service.cc:1467] Processing CreateTablet for tablet 492a05dcabee4d7fa8abde9aabb6e7bc (DEFAULT_TABLE table=test-table [id=9a8d96fd91834f199bf0fb719f92954d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:18:35.694633 7406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 492a05dcabee4d7fa8abde9aabb6e7bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:18:35.707522 7768 tablet_bootstrap.cc:492] T 492a05dcabee4d7fa8abde9aabb6e7bc P 32f91d348a1a4c37bd9c0d703394b172: Bootstrap starting.
I20260929 20:18:35.708748 7767 tablet_bootstrap.cc:492] T 492a05dcabee4d7fa8abde9aabb6e7bc P 6db9cecaaf4a4fc0bf1c8c2b859cf714: Bootstrap starting.
I20260929 20:18:35.711048 7769 tablet_bootstrap.cc:492] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848: Bootstrap starting.
I20260929 20:18:35.712051 7768 tablet_bootstrap.cc:654] T 492a05dcabee4d7fa8abde9aabb6e7bc P 32f91d348a1a4c37bd9c0d703394b172: Neither blocks nor log segments found. Creating new log.
I20260929 20:18:35.712731 7767 tablet_bootstrap.cc:654] T 492a05dcabee4d7fa8abde9aabb6e7bc P 6db9cecaaf4a4fc0bf1c8c2b859cf714: Neither blocks nor log segments found. Creating new log.
I20260929 20:18:35.713393 7768 log.cc:824] T 492a05dcabee4d7fa8abde9aabb6e7bc P 32f91d348a1a4c37bd9c0d703394b172: Log is configured to *not* fsync() on all Append() calls
I20260929 20:18:35.713902 7767 log.cc:824] T 492a05dcabee4d7fa8abde9aabb6e7bc P 6db9cecaaf4a4fc0bf1c8c2b859cf714: Log is configured to *not* fsync() on all Append() calls
I20260929 20:18:35.714790 7769 tablet_bootstrap.cc:654] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848: Neither blocks nor log segments found. Creating new log.
I20260929 20:18:35.716202 7769 log.cc:824] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848: Log is configured to *not* fsync() on all Append() calls
I20260929 20:18:35.718166 7767 tablet_bootstrap.cc:492] T 492a05dcabee4d7fa8abde9aabb6e7bc P 6db9cecaaf4a4fc0bf1c8c2b859cf714: No bootstrap required, opened a new log
I20260929 20:18:35.718457 7767 ts_tablet_manager.cc:1397] T 492a05dcabee4d7fa8abde9aabb6e7bc P 6db9cecaaf4a4fc0bf1c8c2b859cf714: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 20:18:35.719102 7768 tablet_bootstrap.cc:492] T 492a05dcabee4d7fa8abde9aabb6e7bc P 32f91d348a1a4c37bd9c0d703394b172: No bootstrap required, opened a new log
I20260929 20:18:35.719465 7768 ts_tablet_manager.cc:1397] T 492a05dcabee4d7fa8abde9aabb6e7bc P 32f91d348a1a4c37bd9c0d703394b172: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.002s
I20260929 20:18:35.726272 7767 raft_consensus.cc:348] T 492a05dcabee4d7fa8abde9aabb6e7bc P 6db9cecaaf4a4fc0bf1c8c2b859cf714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214e50676fc5411e8f1629f1728c0848" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 41391 } } peers { permanent_uuid: "6db9cecaaf4a4fc0bf1c8c2b859cf714" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 41651 } } peers { permanent_uuid: "32f91d348a1a4c37bd9c0d703394b172" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38437 } }
I20260929 20:18:35.726779 7767 raft_consensus.cc:374] T 492a05dcabee4d7fa8abde9aabb6e7bc P 6db9cecaaf4a4fc0bf1c8c2b859cf714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:18:35.726922 7767 raft_consensus.cc:729] T 492a05dcabee4d7fa8abde9aabb6e7bc P 6db9cecaaf4a4fc0bf1c8c2b859cf714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6db9cecaaf4a4fc0bf1c8c2b859cf714, State: Initialized, Role: FOLLOWER
I20260929 20:18:35.728075 7767 consensus_queue.cc:260] T 492a05dcabee4d7fa8abde9aabb6e7bc P 6db9cecaaf4a4fc0bf1c8c2b859cf714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214e50676fc5411e8f1629f1728c0848" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 41391 } } peers { permanent_uuid: "6db9cecaaf4a4fc0bf1c8c2b859cf714" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 41651 } } peers { permanent_uuid: "32f91d348a1a4c37bd9c0d703394b172" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38437 } }
I20260929 20:18:35.728731 7769 tablet_bootstrap.cc:492] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848: No bootstrap required, opened a new log
I20260929 20:18:35.729118 7769 ts_tablet_manager.cc:1397] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.008s
I20260929 20:18:35.729259 7748 heartbeater.cc:503] Master 127.27.185.126:35331 was elected leader, sending a full tablet report...
I20260929 20:18:35.731353 7767 ts_tablet_manager.cc:1428] T 492a05dcabee4d7fa8abde9aabb6e7bc P 6db9cecaaf4a4fc0bf1c8c2b859cf714: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260929 20:18:35.731303 7768 raft_consensus.cc:348] T 492a05dcabee4d7fa8abde9aabb6e7bc P 32f91d348a1a4c37bd9c0d703394b172 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214e50676fc5411e8f1629f1728c0848" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 41391 } } peers { permanent_uuid: "6db9cecaaf4a4fc0bf1c8c2b859cf714" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 41651 } } peers { permanent_uuid: "32f91d348a1a4c37bd9c0d703394b172" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38437 } }
I20260929 20:18:35.731889 7768 raft_consensus.cc:374] T 492a05dcabee4d7fa8abde9aabb6e7bc P 32f91d348a1a4c37bd9c0d703394b172 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:18:35.731997 7768 raft_consensus.cc:729] T 492a05dcabee4d7fa8abde9aabb6e7bc P 32f91d348a1a4c37bd9c0d703394b172 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32f91d348a1a4c37bd9c0d703394b172, State: Initialized, Role: FOLLOWER
I20260929 20:18:35.732623 7768 consensus_queue.cc:260] T 492a05dcabee4d7fa8abde9aabb6e7bc P 32f91d348a1a4c37bd9c0d703394b172 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214e50676fc5411e8f1629f1728c0848" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 41391 } } peers { permanent_uuid: "6db9cecaaf4a4fc0bf1c8c2b859cf714" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 41651 } } peers { permanent_uuid: "32f91d348a1a4c37bd9c0d703394b172" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38437 } }
I20260929 20:18:35.733839 7614 heartbeater.cc:503] Master 127.27.185.126:35331 was elected leader, sending a full tablet report...
I20260929 20:18:35.734457 7768 ts_tablet_manager.cc:1428] T 492a05dcabee4d7fa8abde9aabb6e7bc P 32f91d348a1a4c37bd9c0d703394b172: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260929 20:18:35.740540 7769 raft_consensus.cc:348] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214e50676fc5411e8f1629f1728c0848" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 41391 } } peers { permanent_uuid: "6db9cecaaf4a4fc0bf1c8c2b859cf714" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 41651 } } peers { permanent_uuid: "32f91d348a1a4c37bd9c0d703394b172" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38437 } }
I20260929 20:18:35.740952 7769 raft_consensus.cc:374] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:18:35.741055 7769 raft_consensus.cc:729] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214e50676fc5411e8f1629f1728c0848, State: Initialized, Role: FOLLOWER
I20260929 20:18:35.741859 7769 consensus_queue.cc:260] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214e50676fc5411e8f1629f1728c0848" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 41391 } } peers { permanent_uuid: "6db9cecaaf4a4fc0bf1c8c2b859cf714" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 41651 } } peers { permanent_uuid: "32f91d348a1a4c37bd9c0d703394b172" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38437 } }
I20260929 20:18:35.742136 7775 raft_consensus.cc:482] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:18:35.742363 7775 raft_consensus.cc:504] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214e50676fc5411e8f1629f1728c0848" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 41391 } } peers { permanent_uuid: "6db9cecaaf4a4fc0bf1c8c2b859cf714" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 41651 } } peers { permanent_uuid: "32f91d348a1a4c37bd9c0d703394b172" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38437 } }
I20260929 20:18:35.745747 7769 ts_tablet_manager.cc:1428] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848: Time spent starting tablet: real 0.016s user 0.008s sys 0.006s
I20260929 20:18:35.745981 7775 leader_election.cc:288] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6db9cecaaf4a4fc0bf1c8c2b859cf714 (127.27.185.67:41651), 32f91d348a1a4c37bd9c0d703394b172 (127.27.185.66:38437)
I20260929 20:18:35.753414 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:18:35.745041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:33225 (local address 127.27.185.67:41651)
0929 20:18:35.745502 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0929 20:18:35.745511 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:18:35.746739 (+ 1228us) server_negotiation.cc:407] Connection header received
0929 20:18:35.746869 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:18:35.746873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:18:35.746955 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:18:35.747105 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:18:35.747744 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.748857 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.751664 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.752465 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.752656 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:18:35.752730 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 20:18:35.753032 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 20:18:35.753156 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279}
I20260929 20:18:35.753453 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:18:35.744489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:41651 (local address 127.27.185.65:33225)
0929 20:18:35.746637 (+ 2148us) negotiation.cc:107] Waiting for socket to connect
0929 20:18:35.746653 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:18:35.746759 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:18:35.747123 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:18:35.747128 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:18:35.747154 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:18:35.747580 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:18:35.747593 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.748993 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.748999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:18:35.751500 (+ 2501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:18:35.751512 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.752671 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.752678 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:18:35.752868 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:18:35.752892 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 20:18:35.752997 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 20:18:35.753128 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2038,"thread_start_us":1706,"threads_started":1}
I20260929 20:18:35.753749 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:18:35.745046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:47063 (local address 127.27.185.66:38437)
0929 20:18:35.745281 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 20:18:35.745290 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:18:35.746627 (+ 1337us) server_negotiation.cc:407] Connection header received
0929 20:18:35.746876 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:18:35.746881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:18:35.746982 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:18:35.747049 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:18:35.748039 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.749072 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.751812 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.752595 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.752777 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:18:35.752860 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 20:18:35.753206 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 20:18:35.753338 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:18:35.754544 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:18:35.744379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:38437 (local address 127.27.185.65:47063)
0929 20:18:35.746493 (+ 2114us) negotiation.cc:107] Waiting for socket to connect
0929 20:18:35.746515 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:18:35.746683 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:18:35.747260 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:18:35.747264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:18:35.747303 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:18:35.747836 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:18:35.747864 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.749205 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.749210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:18:35.751659 (+ 2449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:18:35.751671 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.752827 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.752834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:18:35.753000 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:18:35.753043 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 20:18:35.754073 (+ 1030us) client_negotiation.cc:240] Negotiation successful
0929 20:18:35.754206 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1940,"mutex_wait_us":5,"thread_start_us":2005,"threads_started":1}
I20260929 20:18:35.754576 7703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "492a05dcabee4d7fa8abde9aabb6e7bc" candidate_uuid: "214e50676fc5411e8f1629f1728c0848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6db9cecaaf4a4fc0bf1c8c2b859cf714" is_pre_election: true
I20260929 20:18:35.755157 7703 raft_consensus.cc:2454] T 492a05dcabee4d7fa8abde9aabb6e7bc P 6db9cecaaf4a4fc0bf1c8c2b859cf714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 214e50676fc5411e8f1629f1728c0848 in term 0.
I20260929 20:18:35.755625 7569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "492a05dcabee4d7fa8abde9aabb6e7bc" candidate_uuid: "214e50676fc5411e8f1629f1728c0848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32f91d348a1a4c37bd9c0d703394b172" is_pre_election: true
I20260929 20:18:35.756001 7359 leader_election.cc:302] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 214e50676fc5411e8f1629f1728c0848, 6db9cecaaf4a4fc0bf1c8c2b859cf714; no voters:
I20260929 20:18:35.756083 7569 raft_consensus.cc:2454] T 492a05dcabee4d7fa8abde9aabb6e7bc P 32f91d348a1a4c37bd9c0d703394b172 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 214e50676fc5411e8f1629f1728c0848 in term 0.
I20260929 20:18:35.756377 7775 raft_consensus.cc:2781] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:18:35.756574 7775 raft_consensus.cc:482] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:18:35.756712 7775 raft_consensus.cc:3037] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:18:35.758944 7775 raft_consensus.cc:504] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214e50676fc5411e8f1629f1728c0848" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 41391 } } peers { permanent_uuid: "6db9cecaaf4a4fc0bf1c8c2b859cf714" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 41651 } } peers { permanent_uuid: "32f91d348a1a4c37bd9c0d703394b172" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38437 } }
I20260929 20:18:35.759655 7775 leader_election.cc:288] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848 [CANDIDATE]: Term 1 election: Requested vote from peers 6db9cecaaf4a4fc0bf1c8c2b859cf714 (127.27.185.67:41651), 32f91d348a1a4c37bd9c0d703394b172 (127.27.185.66:38437)
I20260929 20:18:35.760073 7703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "492a05dcabee4d7fa8abde9aabb6e7bc" candidate_uuid: "214e50676fc5411e8f1629f1728c0848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6db9cecaaf4a4fc0bf1c8c2b859cf714"
I20260929 20:18:35.760126 7569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "492a05dcabee4d7fa8abde9aabb6e7bc" candidate_uuid: "214e50676fc5411e8f1629f1728c0848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32f91d348a1a4c37bd9c0d703394b172"
I20260929 20:18:35.760262 7703 raft_consensus.cc:3037] T 492a05dcabee4d7fa8abde9aabb6e7bc P 6db9cecaaf4a4fc0bf1c8c2b859cf714 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:18:35.760337 7569 raft_consensus.cc:3037] T 492a05dcabee4d7fa8abde9aabb6e7bc P 32f91d348a1a4c37bd9c0d703394b172 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:18:35.762504 7703 raft_consensus.cc:2454] T 492a05dcabee4d7fa8abde9aabb6e7bc P 6db9cecaaf4a4fc0bf1c8c2b859cf714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 214e50676fc5411e8f1629f1728c0848 in term 1.
I20260929 20:18:35.762780 7569 raft_consensus.cc:2454] T 492a05dcabee4d7fa8abde9aabb6e7bc P 32f91d348a1a4c37bd9c0d703394b172 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 214e50676fc5411e8f1629f1728c0848 in term 1.
I20260929 20:18:35.763303 7359 leader_election.cc:302] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 214e50676fc5411e8f1629f1728c0848, 6db9cecaaf4a4fc0bf1c8c2b859cf714; no voters:
I20260929 20:18:35.763669 7775 raft_consensus.cc:2781] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:18:35.763854 7775 raft_consensus.cc:686] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848 [term 1 LEADER]: Becoming Leader. State: Replica: 214e50676fc5411e8f1629f1728c0848, State: Running, Role: LEADER
I20260929 20:18:35.764343 7775 consensus_queue.cc:237] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214e50676fc5411e8f1629f1728c0848" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 41391 } } peers { permanent_uuid: "6db9cecaaf4a4fc0bf1c8c2b859cf714" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 41651 } } peers { permanent_uuid: "32f91d348a1a4c37bd9c0d703394b172" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38437 } }
I20260929 20:18:35.769049 7257 catalog_manager.cc:5495] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848 reported cstate change: term changed from 0 to 1, leader changed from <none> to 214e50676fc5411e8f1629f1728c0848 (127.27.185.65). New cstate: current_term: 1 leader_uuid: "214e50676fc5411e8f1629f1728c0848" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214e50676fc5411e8f1629f1728c0848" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 41391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6db9cecaaf4a4fc0bf1c8c2b859cf714" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32f91d348a1a4c37bd9c0d703394b172" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38437 } health_report { overall_health: UNKNOWN } } }
W20260929 20:18:35.774636 7615 tablet.cc:2369] T 492a05dcabee4d7fa8abde9aabb6e7bc P 32f91d348a1a4c37bd9c0d703394b172: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:18:35.786294 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:18:35.780719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58216 (local address 127.27.185.65:41391)
0929 20:18:35.780915 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 20:18:35.780923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:18:35.781023 (+ 100us) server_negotiation.cc:407] Connection header received
0929 20:18:35.781268 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:18:35.781273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:18:35.781358 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:18:35.781532 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:18:35.781887 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.782623 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.784139 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.784553 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.784688 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:18:35.785192 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:18:35.785730 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:18:35.785899 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 20:18:35.785951 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 20:18:35.786034 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:18:35.795158 7703 raft_consensus.cc:1261] T 492a05dcabee4d7fa8abde9aabb6e7bc P 6db9cecaaf4a4fc0bf1c8c2b859cf714 [term 1 FOLLOWER]: Refusing update from remote peer 214e50676fc5411e8f1629f1728c0848: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:18:35.795158 7569 raft_consensus.cc:1261] T 492a05dcabee4d7fa8abde9aabb6e7bc P 32f91d348a1a4c37bd9c0d703394b172 [term 1 FOLLOWER]: Refusing update from remote peer 214e50676fc5411e8f1629f1728c0848: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:18:35.796275 7776 consensus_queue.cc:1035] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6db9cecaaf4a4fc0bf1c8c2b859cf714" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:18:35.796648 7775 consensus_queue.cc:1035] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32f91d348a1a4c37bd9c0d703394b172" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 20:18:35.809959 7475 tablet.cc:2369] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:18:35.810737 7781 mvcc.cc:205] Tried to move back new op lower bound from 7334760922286374912 to 7334760922174582784. Current Snapshot: MvccSnapshot[applied={T|T < 7334760922286374912}]
I20260929 20:18:35.817057 7782 mvcc.cc:205] Tried to move back new op lower bound from 7334760922286374912 to 7334760922174582784. Current Snapshot: MvccSnapshot[applied={T|T < 7334760922286374912}]
I20260929 20:18:35.830431 7257 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:18:35.830780 7257 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:18:35.835667 7257 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 861317cf78fb4e88b4242f1c7f1ba054: Sending DeleteTablet for 3 replicas of tablet 492a05dcabee4d7fa8abde9aabb6e7bc
I20260929 20:18:35.837131 7406 tablet_service.cc:1514] Processing DeleteTablet for tablet 492a05dcabee4d7fa8abde9aabb6e7bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:18:35 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:58212
I20260929 20:18:35.837473 7683 tablet_service.cc:1514] Processing DeleteTablet for tablet 492a05dcabee4d7fa8abde9aabb6e7bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:18:35 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:41340
I20260929 20:18:35.838240 7549 tablet_service.cc:1514] Processing DeleteTablet for tablet 492a05dcabee4d7fa8abde9aabb6e7bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:18:35 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:47040
I20260929 20:18:35.838938 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 7338
I20260929 20:18:35.848097 7793 tablet_replica.cc:330] T 492a05dcabee4d7fa8abde9aabb6e7bc P 6db9cecaaf4a4fc0bf1c8c2b859cf714: stopping tablet replica
I20260929 20:18:35.849587 7793 raft_consensus.cc:2229] T 492a05dcabee4d7fa8abde9aabb6e7bc P 6db9cecaaf4a4fc0bf1c8c2b859cf714 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:18:35.850975 7793 raft_consensus.cc:2258] T 492a05dcabee4d7fa8abde9aabb6e7bc P 6db9cecaaf4a4fc0bf1c8c2b859cf714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:18:35.853709 7794 tablet_replica.cc:330] T 492a05dcabee4d7fa8abde9aabb6e7bc P 32f91d348a1a4c37bd9c0d703394b172: stopping tablet replica
I20260929 20:18:35.854298 7793 ts_tablet_manager.cc:1905] T 492a05dcabee4d7fa8abde9aabb6e7bc P 6db9cecaaf4a4fc0bf1c8c2b859cf714: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:18:35.854259 7794 raft_consensus.cc:2229] T 492a05dcabee4d7fa8abde9aabb6e7bc P 32f91d348a1a4c37bd9c0d703394b172 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:18:35.854822 7794 raft_consensus.cc:2258] T 492a05dcabee4d7fa8abde9aabb6e7bc P 32f91d348a1a4c37bd9c0d703394b172 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:18:35.855584 7792 tablet_replica.cc:330] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848: stopping tablet replica
I20260929 20:18:35.856259 7792 raft_consensus.cc:2229] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:18:35.857088 7792 raft_consensus.cc:2258] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:18:35.858481 7794 ts_tablet_manager.cc:1905] T 492a05dcabee4d7fa8abde9aabb6e7bc P 32f91d348a1a4c37bd9c0d703394b172: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:18:35.863098 7793 ts_tablet_manager.cc:1918] T 492a05dcabee4d7fa8abde9aabb6e7bc P 6db9cecaaf4a4fc0bf1c8c2b859cf714: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:18:35.863350 7793 log.cc:1196] T 492a05dcabee4d7fa8abde9aabb6e7bc P 6db9cecaaf4a4fc0bf1c8c2b859cf714: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-2/wal/wals/492a05dcabee4d7fa8abde9aabb6e7bc
I20260929 20:18:35.863970 7793 ts_tablet_manager.cc:1939] T 492a05dcabee4d7fa8abde9aabb6e7bc P 6db9cecaaf4a4fc0bf1c8c2b859cf714: Deleting consensus metadata
Sep 29 20:18:35 dist-test-slave-4cx3 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.65@KRBTEST.COM
I20260929 20:18:35.865983 7242 catalog_manager.cc:4841] TS 6db9cecaaf4a4fc0bf1c8c2b859cf714 (127.27.185.67:41651): tablet 492a05dcabee4d7fa8abde9aabb6e7bc (table test-table [id=9a8d96fd91834f199bf0fb719f92954d]) successfully deleted
I20260929 20:18:35.866159 7794 ts_tablet_manager.cc:1918] T 492a05dcabee4d7fa8abde9aabb6e7bc P 32f91d348a1a4c37bd9c0d703394b172: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:18:35.866297 7794 log.cc:1196] T 492a05dcabee4d7fa8abde9aabb6e7bc P 32f91d348a1a4c37bd9c0d703394b172: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-1/wal/wals/492a05dcabee4d7fa8abde9aabb6e7bc
I20260929 20:18:35.867851 7794 ts_tablet_manager.cc:1939] T 492a05dcabee4d7fa8abde9aabb6e7bc P 32f91d348a1a4c37bd9c0d703394b172: Deleting consensus metadata
I20260929 20:18:35.868335 7792 ts_tablet_manager.cc:1905] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:18:35.869675 7244 catalog_manager.cc:4841] TS 32f91d348a1a4c37bd9c0d703394b172 (127.27.185.66:38437): tablet 492a05dcabee4d7fa8abde9aabb6e7bc (table test-table [id=9a8d96fd91834f199bf0fb719f92954d]) successfully deleted
I20260929 20:18:35.874534 7792 ts_tablet_manager.cc:1918] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:18:35.874720 7792 log.cc:1196] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790712966917858-28389-0/minicluster-data/ts-0/wal/wals/492a05dcabee4d7fa8abde9aabb6e7bc
I20260929 20:18:35.874945 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:18:35.840208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58232 (local address 127.27.185.65:41391)
0929 20:18:35.840382 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 20:18:35.840388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:18:35.850959 (+ 10571us) server_negotiation.cc:407] Connection header received
0929 20:18:35.851495 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:18:35.851502 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:18:35.851591 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:18:35.851691 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:18:35.854454 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.855895 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.858267 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.859066 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.859184 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:18:35.865978 (+ 6794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:18:35.865997 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:18:35.866018 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:18:35.866068 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:18:35.870781 (+ 4713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:18:35.872021 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:18:35.872031 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:18:35.872035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:18:35.872212 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:18:35.872732 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:18:35.872738 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:18:35.872746 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:18:35.873301 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:18:35.873630 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0929 20:18:35.874254 (+ 624us) server_negotiation.cc:299] Negotiation successful
0929 20:18:35.874537 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:18:35.875456 7792 ts_tablet_manager.cc:1939] T 492a05dcabee4d7fa8abde9aabb6e7bc P 214e50676fc5411e8f1629f1728c0848: Deleting consensus metadata
I20260929 20:18:35.876240 7466 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:18:35.975701 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 7338
I20260929 20:18:35.975807 7245 catalog_manager.cc:4841] TS 214e50676fc5411e8f1629f1728c0848 (127.27.185.65:41391): tablet 492a05dcabee4d7fa8abde9aabb6e7bc (table test-table [id=9a8d96fd91834f199bf0fb719f92954d]) successfully deleted
I20260929 20:18:35.994629 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 7478
Sep 29 20:18:36 dist-test-slave-4cx3 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.66@KRBTEST.COM
I20260929 20:18:36.010519 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:18:35.995419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47048 (local address 127.27.185.66:38437)
0929 20:18:35.995614 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 20:18:35.995620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:18:35.995724 (+ 104us) server_negotiation.cc:407] Connection header received
0929 20:18:35.995951 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:18:35.995955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:18:35.996029 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:18:35.996147 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:18:35.997389 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:18:35.998217 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:18:35.999632 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:18:36.000030 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:18:36.000168 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:18:36.004680 (+ 4512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:18:36.004695 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:18:36.004712 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:18:36.004752 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:18:36.007626 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:18:36.008350 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:18:36.008357 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:18:36.008359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:18:36.008482 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:18:36.008886 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:18:36.008890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:18:36.008892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:18:36.009285 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:18:36.009602 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0929 20:18:36.009935 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 20:18:36.010172 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:18:36.011222 7609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:18:36.103171 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 7478
I20260929 20:18:36.120673 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 7620
Sep 29 20:18:36 dist-test-slave-4cx3 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.67@KRBTEST.COM
I20260929 20:18:36.136771 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:18:36.121408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41344 (local address 127.27.185.67:41651)
0929 20:18:36.121609 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 20:18:36.121617 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:18:36.121639 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:18:36.121861 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:18:36.121867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:18:36.121952 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:18:36.122075 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:18:36.123616 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:18:36.124407 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:18:36.125728 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:18:36.126126 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:18:36.126264 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:18:36.131032 (+ 4768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:18:36.131049 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:18:36.131065 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:18:36.131100 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:18:36.133788 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:18:36.134558 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:18:36.134566 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:18:36.134568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:18:36.134690 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:18:36.135100 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:18:36.135106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:18:36.135109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:18:36.135573 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:18:36.135888 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0929 20:18:36.136193 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 20:18:36.136424 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:18:36.137547 7743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:18:36.228138 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 7620
I20260929 20:18:36.245003 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 7226
I20260929 20:18:36.246286 7288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:18:36.360051 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 7226
I20260929 20:18:36.377758 28389 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 20:18:36.713259 28389 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 20:18:36.713569 28389 mini_ranger.cc:66] Stopping Ranger...
I20260929 20:18:37.059731 28389 mini_ranger.cc:68] Stopped Ranger
2026-09-29 20:18:37.060 UTC [6812] LOG: received smart shutdown request
2026-09-29 20:18:37.064 UTC [6812] LOG: background worker "logical replication launcher" (PID 6821) exited with exit code 1
2026-09-29 20:18:37.064 UTC [6816] LOG: shutting down
2026-09-29 20:18:37.265 UTC [6812] LOG: database system is shut down
2026-09-29T20:18:37Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49429 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:18:37 dist-test-slave-4cx3 krb5kdc[7808](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:18:37 dist-test-slave-4cx3 krb5kdc[7808](info): set up 2 sockets
Sep 29 20:18:37 dist-test-slave-4cx3 krb5kdc[7808](info): commencing operation
krb5kdc: starting...
W20260929 20:18:38.530803 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:18:38 dist-test-slave-4cx3 krb5kdc[7808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:18:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:18:38Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.185.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.185.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.185.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/rangeradmin_127.27.185.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.185.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/rangeradmin_127.27.185.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.185.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.185.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.185.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/rangerlookup_127.27.185.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.185.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/rangerlookup_127.27.185.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.185.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.185.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.185.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/HTTP_127.27.185.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.185.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/HTTP_127.27.185.84@KRBTEST.COM.keytab.
I20260929 20:18:38.870910 28389 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskPFKKvj/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/postgres -l logfile start
2026-09-29 20:18:41.602 UTC [7838] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 20:18:41.602 UTC [7838] LOG: listening on IPv4 address "127.27.185.84", port 54967
2026-09-29 20:18:41.606 UTC [7838] LOG: listening on Unix socket "/tmp/.s.PGSQL.54967"
2026-09-29 20:18:41.629 UTC [7841] LOG: database system was shut down at 2026-09-29 20:18:39 UTC
2026-09-29 20:18:41.633 UTC [7838] LOG: database system is ready to accept connections
I20260929 20:18:42.606679 28389 mini_postgres.cc:96] Postgres bound to 54967
2026-09-29 20:18:42.624 UTC [7849] FATAL: database "slave" does not exist
127.27.185.84:54967 - accepting connections
I20260929 20:18:42.624902 28389 mini_ranger.cc:161] Starting Ranger...
I20260929 20:18:42.651782 28389 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 20:18:43.215373 28389 mini_ranger.cc:87] Created ranger Postgres database
I20260929 20:18:43.215541 28389 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ranger-admin
2026-09-29 20:18:43,498 [I] DB FLAVOR :POSTGRES
2026-09-29 20:18:43,499 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 20:18:43,499 [I] Checking connection..
2026-09-29 20:18:43,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:54967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 20:18:43,777 [I] Checking connection passed.
2026-09-29 20:18:43,777 [I] --------- Verifying version history table ---------
2026-09-29 20:18:43,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:54967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 20:18:44,017 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 20:18:44,017 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:54967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 20:18:44,266 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 20:18:44,266 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 20:18:44,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:54967/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 20:18:44.513 UTC [7913] WARNING: there is no transaction in progress
2026-09-29 20:18:44,520 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 20:18:44,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:54967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 20:18:44,784 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 20:18:44,784 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 20:18:44,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:54967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 20:18:45,029 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:54967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 20:18:45,275 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:54967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-4cx3.c.gcp-upstream.internal','N') ;"
2026-09-29 20:18:45,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:54967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 20:18:45,761 [I] Table x_portal_user does not exist in database ranger
2026-09-29 20:18:45,762 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:54967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 20:18:46,029 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 20:18:46,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:54967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 20:18:46,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:54967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 20:18:46,512 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 20:18:46,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:54967/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 20:18:47.543 UTC [8021] WARNING: there is no transaction in progress
2026-09-29 20:18:47.561 UTC [8021] WARNING: there is no transaction in progress
2026-09-29 20:18:47.579 UTC [8021] WARNING: there is no transaction in progress
2026-09-29 20:18:47.597 UTC [8021] WARNING: there is no transaction in progress
2026-09-29 20:18:47.614 UTC [8021] WARNING: there is no transaction in progress
2026-09-29 20:18:47.632 UTC [8021] WARNING: there is no transaction in progress
2026-09-29 20:18:47.692 UTC [8021] WARNING: there is no transaction in progress
2026-09-29 20:18:47.699 UTC [8021] WARNING: there is no transaction in progress
2026-09-29 20:18:47.713 UTC [8021] WARNING: there is no transaction in progress
2026-09-29 20:18:47.724 UTC [8021] WARNING: there is no transaction in progress
2026-09-29 20:18:47.734 UTC [8021] WARNING: there is no transaction in progress
2026-09-29 20:18:47.748 UTC [8021] WARNING: there is no transaction in progress
2026-09-29 20:18:47.758 UTC [8021] WARNING: there is no transaction in progress
2026-09-29 20:18:47.765 UTC [8021] WARNING: there is no transaction in progress
2026-09-29 20:18:47.772 UTC [8021] WARNING: there is no transaction in progress
2026-09-29 20:18:47.780 UTC [8021] WARNING: there is no transaction in progress
2026-09-29 20:18:48,484 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 20:18:48,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:54967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 20:18:48,716 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 20:18:48,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:54967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 20:18:48,977 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 20:18:48,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:54967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 20:18:49,227 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 20:18:49,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:54967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 20:18:49,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:54967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 20:18:49,702 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:54967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-4cx3.c.gcp-upstream.internal';"
2026-09-29 20:18:49,926 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 20:18:49,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:54967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 20:18:50,176 [I] DB_PATCHES have already been applied
I20260929 20:18:50.182667 28389 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ranger-admin:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskPFKKvj/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 20:18:50.182789 28389 mini_ranger.cc:193] Using host: 127.27.185.84
I20260929 20:18:50.197372 28389 mini_ranger.cc:227] Ranger admin URL: http://127.27.185.84:50379
Sep 29, 2026 8:18:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 8:18:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 8:18:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 8:18:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 8:18:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 8:18:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 8:18:51 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 20:18:51.359097 28389 mini_ranger.cc:160] Time spent starting Ranger: real 8.734s user 0.007s sys 0.033s
Sep 29, 2026 8:19:11 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 8:19:11 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 8:19:11 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 8:19:11 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 8:19:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 8:19:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 8:19:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 8:19:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 8:19:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 8:19:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 8:19:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 8:19:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 8:19:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 8:19:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 8:19:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 8:19:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 8:19:11 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 8:19:12 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 20:19:13.154309 28389 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.185.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.185.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.185.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/rangerkms_127.27.185.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.185.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/rangerkms_127.27.185.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.185.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.185.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.185.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/HTTP_127.27.185.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.185.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/HTTP_127.27.185.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 20:19:13 dist-test-slave-4cx3 krb5kdc[7808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713153, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 20:19:13.324519 28389 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 20:19:13.348133 28389 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 20:19:14.332078 28389 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 20:19:14.332203 28389 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ranger-kms
I20260929 20:19:14.332248 28389 mini_ranger_kms.cc:227] Using postgres at 127.27.185.84:54967
2026-09-29 20:19:14,697 [I] DB FLAVOR :POSTGRES
2026-09-29 20:19:14,697 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 20:19:14,697 [I] Checking connection
2026-09-29 20:19:14,697 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:54967/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 20:19:14,949 [I] connection success
2026-09-29 20:19:14,950 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 20:19:14,950 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 20:19:14,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:54967/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 20:19:15,179 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 20:19:15,179 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 20:19:15,179 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 20:19:15,180 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.185.84:54967/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 20:19:15,448 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 20:19:15.962925 28389 mini_ranger_kms.cc:310] Created kms service
I20260929 20:19:16.180277 28389 mini_ranger_kms.cc:326] Created kudu user
I20260929 20:19:16.229454 28389 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 20:19:16 dist-test-slave-4cx3 krb5kdc[7808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713153, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.185.84@KRBTEST.COM
I20260929 20:19:16.652697 28389 mini_ranger_kms.cc:384] Added ranger policy
I20260929 20:19:16.653090 28389 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ranger-kms:/tmp/dist-test-taskPFKKvj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskPFKKvj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 20:19:16.653157 28389 mini_ranger_kms.cc:240] Using host: 127.27.185.84
I20260929 20:19:16.671903 28389 mini_ranger_kms.cc:276] Ranger KMS PID: 8201
I20260929 20:19:16.672062 28389 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.185.84:45025
Sep 29, 2026 8:19:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 8:19:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 8:19:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 8:19:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 8:19:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 8:19:18 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 20:19:18.967238 28389 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.643s user 0.080s sys 0.304s
I20260929 20:19:18.967515 28389 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 20:19:18.967705 28389 mini_ranger_kms.cc:401] 127.27.185.84:45025/kms/v1/keys
Sep 29 20:19:19 dist-test-slave-4cx3 krb5kdc[7808](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790713159, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.185.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 20:19:19 dist-test-slave-4cx3 krb5kdc[7808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713159, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.185.84@KRBTEST.COM for HTTP/127.27.185.84@KRBTEST.COM
Sep 29 20:19:19 dist-test-slave-4cx3 krb5kdc[7808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713159, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.185.84@KRBTEST.COM for HTTP/127.27.185.84@KRBTEST.COM
Sep 29 20:19:22 dist-test-slave-4cx3 krb5kdc[7808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
I20260929 20:19:22.236398 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:35879
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:37481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:35879
--ranger_config_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.185.84:45025/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:19:22.593793 8252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:19:22.594157 8252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:19:22.594309 8252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:19:22.601788 8252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:19:22.601943 8252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:19:22.602054 8252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:19:22.602159 8252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:19:22.602239 8252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:19:22.613638 8252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:37481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.185.84:45025/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:35879
--ranger_config_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:35879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:19:22.615130 8252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:19:22.616642 8252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:19:22.630525 8257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:22.630501 8258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:22.631461 8260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:19:22.631484 8252 server_base.cc:1034] running on GCE node
I20260929 20:19:22.632488 8252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:19:22.634227 8252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:19:22.635515 8252 hybrid_clock.cc:648] HybridClock initialized: now 1790713162635452 us; error 55 us; skew 500 ppm
I20260929 20:19:22.635948 8252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:19:22 dist-test-slave-4cx3 krb5kdc[7808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713162, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:19:22.643728 8252 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:19:22.645543 8252 webserver.cc:460] Webserver started at http://127.27.185.126:45281/ using document root <none> and password file <none>
I20260929 20:19:22.646263 8252 fs_manager.cc:359] Metadata directory not provided
I20260929 20:19:22.646415 8252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:19:22.646739 8252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:19:22 dist-test-slave-4cx3 krb5kdc[7808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713162, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for HTTP/127.27.185.84@KRBTEST.COM
I20260929 20:19:22.696264 8252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "3d123993bcaf4172a8045f5e988582b6"
format_stamp: "Formatted at 2026-09-29 20:19:22 on dist-test-slave-4cx3"
server_key: "ed97eea3673fb12e5d7aebf376d5c20e"
server_key_iv: "88075ac467e8abf6e355617f1906a24c"
server_key_version: "kuduclusterkey@0"
I20260929 20:19:22.697212 8252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "3d123993bcaf4172a8045f5e988582b6"
format_stamp: "Formatted at 2026-09-29 20:19:22 on dist-test-slave-4cx3"
server_key: "ed97eea3673fb12e5d7aebf376d5c20e"
server_key_iv: "88075ac467e8abf6e355617f1906a24c"
server_key_version: "kuduclusterkey@0"
I20260929 20:19:22.702875 8252 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:19:22.719856 8267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:22.721486 8252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 20:19:22.721679 8252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "3d123993bcaf4172a8045f5e988582b6"
format_stamp: "Formatted at 2026-09-29 20:19:22 on dist-test-slave-4cx3"
server_key: "ed97eea3673fb12e5d7aebf376d5c20e"
server_key_iv: "88075ac467e8abf6e355617f1906a24c"
server_key_version: "kuduclusterkey@0"
I20260929 20:19:22.721904 8252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:19:22.750802 8252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:19:22.756249 8252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:19:22.756614 8252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:19:22.780166 8252 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:35879
I20260929 20:19:22.780211 8318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:35879 every 8 connection(s)
I20260929 20:19:22.781621 8252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:19:22.784271 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 8252
I20260929 20:19:22.784512 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:19:22.785197 28389 external_mini_cluster.cc:1428] Setting key c7bdc4894d159b047750c1d95cffe824
I20260929 20:19:22.786450 8319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:19:22.801492 8319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6: Bootstrap starting.
I20260929 20:19:22.806073 8319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6: Neither blocks nor log segments found. Creating new log.
I20260929 20:19:22.807515 8319 log.cc:824] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6: Log is configured to *not* fsync() on all Append() calls
I20260929 20:19:22.811185 8319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6: No bootstrap required, opened a new log
Sep 29 20:19:22 dist-test-slave-4cx3 krb5kdc[7808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:19:22.821717 8319 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d123993bcaf4172a8045f5e988582b6" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 35879 } }
I20260929 20:19:22.822275 8319 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:19:22.822441 8319 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d123993bcaf4172a8045f5e988582b6, State: Initialized, Role: FOLLOWER
I20260929 20:19:22.823231 8319 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d123993bcaf4172a8045f5e988582b6" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 35879 } }
I20260929 20:19:22.823591 8319 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:19:22.823777 8319 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:19:22.823982 8319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:19:22.827159 8319 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d123993bcaf4172a8045f5e988582b6" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 35879 } }
I20260929 20:19:22.827824 8319 leader_election.cc:302] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d123993bcaf4172a8045f5e988582b6; no voters:
I20260929 20:19:22.828570 8319 leader_election.cc:288] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:19:22.828707 8324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:19:22.830030 8322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:22.787498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43538 (local address 127.27.185.126:35879)
0929 20:19:22.801974 (+ 14476us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:22.801993 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:22.802049 (+ 56us) server_negotiation.cc:407] Connection header received
0929 20:19:22.803981 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:22.804004 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:22.804438 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:22.804856 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:22.806348 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:22.807833 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:22.809607 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:22.810300 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:22.810521 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:22.820265 (+ 9744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:22.820275 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:22.820295 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:22.820340 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:22.824225 (+ 3885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:22.825036 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:22.825049 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:22.825059 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:22.825268 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:22.825738 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:22.825744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:22.825747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:22.826478 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:22.826927 (+ 449us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:22.827870 (+ 943us) server_negotiation.cc:299] Negotiation successful
0929 20:19:22.828271 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14187,"thread_start_us":15816,"threads_started":1}
I20260929 20:19:22.835762 8324 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [term 1 LEADER]: Becoming Leader. State: Replica: 3d123993bcaf4172a8045f5e988582b6, State: Running, Role: LEADER
I20260929 20:19:22.836431 8324 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d123993bcaf4172a8045f5e988582b6" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 35879 } }
I20260929 20:19:22.837276 8319 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:19:22.839324 8326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d123993bcaf4172a8045f5e988582b6. Latest consensus state: current_term: 1 leader_uuid: "3d123993bcaf4172a8045f5e988582b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d123993bcaf4172a8045f5e988582b6" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 35879 } } }
I20260929 20:19:22.839730 8326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [sys.catalog]: This master's current role is: LEADER
I20260929 20:19:22.840860 8325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d123993bcaf4172a8045f5e988582b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d123993bcaf4172a8045f5e988582b6" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 35879 } } }
I20260929 20:19:22.841153 8325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [sys.catalog]: This master's current role is: LEADER
I20260929 20:19:22.850069 8330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:19:22.862290 8330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:19:22.874996 8330 catalog_manager.cc:1347] Generated new cluster ID: 68aeddefdcbd41d58dc8e11598ef9c8c
I20260929 20:19:22.875222 8330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:19:22.877938 8319 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 20:19:22.907181 8330 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:19:22.909996 8330 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:19:22.927855 8330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6: Generated new TSK 0
I20260929 20:19:22.929140 8330 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
I20260929 20:19:23.755462 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.185.84:45025/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:0
--local_ip_for_outbound_sockets=127.27.185.65
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:35879
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:37481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:19:23 dist-test-slave-4cx3 krb5kdc[7808](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790713163, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 20:19:24.216338 8364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:19:24.216777 8364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:19:24.216998 8364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:19:24.225126 8364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:19:24.225361 8364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:19:24.225620 8364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:19:24.239705 8364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:37481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.185.84:45025/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:35879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:19:24.241642 8364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:19:24.243419 8364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:19:24.260588 8374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:24.261205 8375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:19:24.275256 8364 server_base.cc:1034] running on GCE node
W20260929 20:19:24.276538 8377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:19:24.277485 8364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:19:24.278869 8364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:19:24.280186 8364 hybrid_clock.cc:648] HybridClock initialized: now 1790713164280104 us; error 53 us; skew 500 ppm
I20260929 20:19:24.280588 8364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:19:24 dist-test-slave-4cx3 krb5kdc[7808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713164, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:19:24.285136 8364 init.cc:377] Logged in from keytab as kudu/127.27.185.65@KRBTEST.COM (short username kudu)
I20260929 20:19:24.296918 8364 webserver.cc:460] Webserver started at http://127.27.185.65:42445/ using document root <none> and password file <none>
I20260929 20:19:24.297927 8364 fs_manager.cc:359] Metadata directory not provided
I20260929 20:19:24.298149 8364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:19:24.298583 8364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:19:24 dist-test-slave-4cx3 krb5kdc[7808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713164, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for HTTP/127.27.185.84@KRBTEST.COM
I20260929 20:19:24.319429 8364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/data/instance:
uuid: "9a70c4b2e1764708a9dd9c07c329f9a3"
format_stamp: "Formatted at 2026-09-29 20:19:24 on dist-test-slave-4cx3"
server_key: "d23cc0db5f88a5787435718ed237a655"
server_key_iv: "0f0860acc672126dc069d683d3e5c20f"
server_key_version: "kuduclusterkey@0"
I20260929 20:19:24.320680 8364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance:
uuid: "9a70c4b2e1764708a9dd9c07c329f9a3"
format_stamp: "Formatted at 2026-09-29 20:19:24 on dist-test-slave-4cx3"
server_key: "d23cc0db5f88a5787435718ed237a655"
server_key_iv: "0f0860acc672126dc069d683d3e5c20f"
server_key_version: "kuduclusterkey@0"
I20260929 20:19:24.341380 8364 fs_manager.cc:691] Time spent creating directory manager: real 0.020s user 0.013s sys 0.004s
I20260929 20:19:24.364102 8384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:24.366194 8364 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260929 20:19:24.366472 8364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "9a70c4b2e1764708a9dd9c07c329f9a3"
format_stamp: "Formatted at 2026-09-29 20:19:24 on dist-test-slave-4cx3"
server_key: "d23cc0db5f88a5787435718ed237a655"
server_key_iv: "0f0860acc672126dc069d683d3e5c20f"
server_key_version: "kuduclusterkey@0"
I20260929 20:19:24.366814 8364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:19:24.416458 8364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:19:24.422106 8364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
Sep 29 20:19:24 dist-test-slave-4cx3 krb5kdc[7808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713163, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for HTTP/127.27.185.84@KRBTEST.COM
I20260929 20:19:24.423148 8364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:19:24.424548 8364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:19:24.433856 8364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:19:24.433990 8364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:24.434171 8364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:19:24.434252 8364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:24.584344 8364 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:32983
I20260929 20:19:24.584350 8502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:32983 every 8 connection(s)
I20260929 20:19:24.585866 8364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:19:24.591818 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 8364
I20260929 20:19:24.592125 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance
I20260929 20:19:24.593246 28389 external_mini_cluster.cc:1428] Setting key f816eaf175a28f525e1f5ba4f81d8c7f
Sep 29 20:19:24 dist-test-slave-4cx3 krb5kdc[7808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713164, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.185.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.66@KRBTEST.COM" created.
I20260929 20:19:24.626209 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:24.589421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:34211 (local address 127.27.185.126:35879)
0929 20:19:24.599783 (+ 10362us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:24.599789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:24.603445 (+ 3656us) server_negotiation.cc:407] Connection header received
0929 20:19:24.603859 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:24.603865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:24.603948 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:24.604024 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:24.606243 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:24.607484 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:24.610078 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:24.610725 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:24.610881 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:24.615803 (+ 4922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:24.615811 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:24.615816 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:24.615862 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:24.619630 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:24.620714 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:24.620720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:24.620723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:24.620893 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:24.621489 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:24.621494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:24.621498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:24.621881 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:24.622038 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:24.625583 (+ 3545us) server_negotiation.cc:299] Negotiation successful
0929 20:19:24.625842 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10223,"thread_start_us":12238,"threads_started":1}
I20260929 20:19:24.625289 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:24.590242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:35879 (local address 127.27.185.65:34211)
0929 20:19:24.600985 (+ 10743us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:24.601041 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:24.603535 (+ 2494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:24.604597 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:24.604609 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:24.605212 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:19:24.606033 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:24.606056 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:24.607697 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:24.607704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:24.609880 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:24.609905 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:24.611223 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:24.611231 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:24.611580 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:24.612375 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:19:24.612410 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:19:24.615466 (+ 3056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:19:24.619852 (+ 4386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:24.619862 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:24.619883 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:24.620547 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:24.621043 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:24.621048 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:24.621051 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:24.621339 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:24.622125 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:19:24.622134 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:19:24.622741 (+ 607us) client_negotiation.cc:769] Sending connection context
0929 20:19:24.623108 (+ 367us) client_negotiation.cc:240] Negotiation successful
0929 20:19:24.623733 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10415,"thread_start_us":10198,"threads_started":1}
I20260929 20:19:24.628924 8503 heartbeater.cc:347] Connected to a master server at 127.27.185.126:35879
I20260929 20:19:24.629288 8503 heartbeater.cc:464] Registering TS with master...
I20260929 20:19:24.630244 8503 heartbeater.cc:511] Master 127.27.185.126:35879 requested a full tablet report, sending...
I20260929 20:19:24.633034 8284 ts_manager.cc:194] Registered new tserver with Master: 9a70c4b2e1764708a9dd9c07c329f9a3 (127.27.185.65:32983)
I20260929 20:19:24.635085 8284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.65@KRBTEST.COM'} at 127.27.185.65:34211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
I20260929 20:19:24.665179 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.185.84:45025/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:0
--local_ip_for_outbound_sockets=127.27.185.66
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:35879
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:37481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:19:24 dist-test-slave-4cx3 krb5kdc[7808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713163, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for HTTP/127.27.185.84@KRBTEST.COM
W20260929 20:19:25.132947 8509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:19:25.133350 8509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:19:25.133496 8509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:19:25.144402 8509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:19:25.144560 8509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:19:25.144719 8509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:19:25.159597 8509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:37481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.185.84:45025/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:35879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:19:25.161365 8509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:19:25.163060 8509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:19:25.179548 8523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:25.179908 8524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:25.180760 8526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:19:25.181167 8509 server_base.cc:1034] running on GCE node
I20260929 20:19:25.181880 8509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:19:25.182904 8509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:19:25.184162 8509 hybrid_clock.cc:648] HybridClock initialized: now 1790713165184097 us; error 51 us; skew 500 ppm
I20260929 20:19:25.184537 8509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:19:25 dist-test-slave-4cx3 krb5kdc[7808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713165, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:19:25.188134 8509 init.cc:377] Logged in from keytab as kudu/127.27.185.66@KRBTEST.COM (short username kudu)
I20260929 20:19:25.189718 8509 webserver.cc:460] Webserver started at http://127.27.185.66:34441/ using document root <none> and password file <none>
I20260929 20:19:25.190431 8509 fs_manager.cc:359] Metadata directory not provided
I20260929 20:19:25.190574 8509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:19:25.190887 8509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:19:25 dist-test-slave-4cx3 krb5kdc[7808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713165, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for HTTP/127.27.185.84@KRBTEST.COM
I20260929 20:19:25.207964 8509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/data/instance:
uuid: "a588c33cea4a4a939a3c6ee7b0f85bb3"
format_stamp: "Formatted at 2026-09-29 20:19:25 on dist-test-slave-4cx3"
server_key: "e84bcba5dbfff4b962469f56cd08bd05"
server_key_iv: "c06918f558e7f2256b9ed1c832f9ba4d"
server_key_version: "kuduclusterkey@0"
I20260929 20:19:25.208828 8509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance:
uuid: "a588c33cea4a4a939a3c6ee7b0f85bb3"
format_stamp: "Formatted at 2026-09-29 20:19:25 on dist-test-slave-4cx3"
server_key: "e84bcba5dbfff4b962469f56cd08bd05"
server_key_iv: "c06918f558e7f2256b9ed1c832f9ba4d"
server_key_version: "kuduclusterkey@0"
I20260929 20:19:25.214363 8509 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 20:19:25.232012 8533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:25.233950 8509 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 20:19:25.234165 8509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "a588c33cea4a4a939a3c6ee7b0f85bb3"
format_stamp: "Formatted at 2026-09-29 20:19:25 on dist-test-slave-4cx3"
server_key: "e84bcba5dbfff4b962469f56cd08bd05"
server_key_iv: "c06918f558e7f2256b9ed1c832f9ba4d"
server_key_version: "kuduclusterkey@0"
I20260929 20:19:25.234454 8509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:19:25.258227 8509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:19:25.262470 8509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:19:25.262792 8509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:19:25.263939 8509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:19:25.265492 8509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:19:25.265596 8509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:25.265733 8509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:19:25.265806 8509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:25.312527 8509 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:35725
I20260929 20:19:25.312544 8645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:35725 every 8 connection(s)
I20260929 20:19:25.313938 8509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:19:25.320161 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 8509
I20260929 20:19:25.320379 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance
I20260929 20:19:25.321190 28389 external_mini_cluster.cc:1428] Setting key c261e18ff1d5de93486cb57ce722972f
Sep 29 20:19:25 dist-test-slave-4cx3 krb5kdc[7808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713165, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.185.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.67@KRBTEST.COM" created.
I20260929 20:19:25.373798 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:25.317243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:59819 (local address 127.27.185.126:35879)
0929 20:19:25.336070 (+ 18827us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:25.336078 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:25.353063 (+ 16985us) server_negotiation.cc:407] Connection header received
0929 20:19:25.355229 (+ 2166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:25.355236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:25.355336 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:25.355507 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:25.357782 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:25.358967 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:25.361523 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:25.362125 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:25.362232 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:25.366385 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:25.366390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:25.366393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:25.366430 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:25.369405 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:25.370430 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:25.370434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:25.370437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:25.370641 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:25.371249 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:25.371255 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:25.371258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:25.371658 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:25.371812 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:25.372849 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0929 20:19:25.373142 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18639,"thread_start_us":18831,"threads_started":1}
I20260929 20:19:25.374887 8649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:25.318167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:35879 (local address 127.27.185.66:59819)
0929 20:19:25.352124 (+ 33957us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:25.352196 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:25.354829 (+ 2633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:25.355993 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:25.356007 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:25.356665 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:19:25.357541 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:25.357569 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:25.359183 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:25.359188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:25.361315 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:25.361331 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:25.362611 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:25.362620 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:25.362942 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:25.363804 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:19:25.363828 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:19:25.366116 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:19:25.369602 (+ 3486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:25.369612 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:25.369633 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:25.370296 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:25.370804 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:25.370809 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:25.370813 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:25.371100 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:25.371888 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:19:25.371896 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:19:25.372364 (+ 468us) client_negotiation.cc:769] Sending connection context
0929 20:19:25.372690 (+ 326us) client_negotiation.cc:240] Negotiation successful
0929 20:19:25.373269 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33544,"thread_start_us":33275,"threads_started":1}
I20260929 20:19:25.376541 8646 heartbeater.cc:347] Connected to a master server at 127.27.185.126:35879
I20260929 20:19:25.376883 8646 heartbeater.cc:464] Registering TS with master...
I20260929 20:19:25.377748 8646 heartbeater.cc:511] Master 127.27.185.126:35879 requested a full tablet report, sending...
I20260929 20:19:25.379627 8284 ts_manager.cc:194] Registered new tserver with Master: a588c33cea4a4a939a3c6ee7b0f85bb3 (127.27.185.66:35725)
I20260929 20:19:25.380724 8284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.66@KRBTEST.COM'} at 127.27.185.66:59819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
I20260929 20:19:25.411146 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.185.84:45025/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:0
--local_ip_for_outbound_sockets=127.27.185.67
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:35879
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:37481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:19:25.639170 8503 heartbeater.cc:503] Master 127.27.185.126:35879 was elected leader, sending a full tablet report...
W20260929 20:19:25.761279 8652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:19:25.761562 8652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:19:25.761714 8652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:19:25.768987 8652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:19:25.769094 8652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:19:25.769258 8652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:19:25.779156 8652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:37481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.185.84:45025/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:35879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:19:25.780596 8652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:19:25.781848 8652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:19:25.795627 8657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:25.795593 8658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:25.795526 8660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:19:25.796355 8652 server_base.cc:1034] running on GCE node
I20260929 20:19:25.797070 8652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:19:25.798017 8652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:19:25.799221 8652 hybrid_clock.cc:648] HybridClock initialized: now 1790713165799169 us; error 44 us; skew 500 ppm
I20260929 20:19:25.799605 8652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:19:25 dist-test-slave-4cx3 krb5kdc[7808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713165, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:19:25.802717 8652 init.cc:377] Logged in from keytab as kudu/127.27.185.67@KRBTEST.COM (short username kudu)
I20260929 20:19:25.804317 8652 webserver.cc:460] Webserver started at http://127.27.185.67:45985/ using document root <none> and password file <none>
I20260929 20:19:25.804970 8652 fs_manager.cc:359] Metadata directory not provided
I20260929 20:19:25.805069 8652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:19:25.805317 8652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:19:25 dist-test-slave-4cx3 krb5kdc[7808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713165, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for HTTP/127.27.185.84@KRBTEST.COM
I20260929 20:19:25.821765 8652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/data/instance:
uuid: "3bef4fa96cd24026bc828f3b6d324eba"
format_stamp: "Formatted at 2026-09-29 20:19:25 on dist-test-slave-4cx3"
server_key: "65b9457741a967eaf81de9ad961dc7db"
server_key_iv: "9cf8e45d77be1d3b3db35bde2a7a75de"
server_key_version: "kuduclusterkey@0"
I20260929 20:19:25.822968 8652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance:
uuid: "3bef4fa96cd24026bc828f3b6d324eba"
format_stamp: "Formatted at 2026-09-29 20:19:25 on dist-test-slave-4cx3"
server_key: "65b9457741a967eaf81de9ad961dc7db"
server_key_iv: "9cf8e45d77be1d3b3db35bde2a7a75de"
server_key_version: "kuduclusterkey@0"
I20260929 20:19:25.829208 8652 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 20:19:25.845533 8667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:25.847054 8652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:19:25.847239 8652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "3bef4fa96cd24026bc828f3b6d324eba"
format_stamp: "Formatted at 2026-09-29 20:19:25 on dist-test-slave-4cx3"
server_key: "65b9457741a967eaf81de9ad961dc7db"
server_key_iv: "9cf8e45d77be1d3b3db35bde2a7a75de"
server_key_version: "kuduclusterkey@0"
I20260929 20:19:25.847528 8652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:19:25.866894 8652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:19:25.870905 8652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:19:25.871243 8652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:19:25.872262 8652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:19:25.873775 8652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:19:25.873884 8652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:25.874034 8652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:19:25.874105 8652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:25.907619 8652 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:45287
I20260929 20:19:25.907667 8779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:45287 every 8 connection(s)
I20260929 20:19:25.908947 8652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:19:25.918345 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 8652
I20260929 20:19:25.918591 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance
I20260929 20:19:25.919073 28389 external_mini_cluster.cc:1428] Setting key 4f936f5d6b834dc0d237c387bc37edf1
Sep 29 20:19:25 dist-test-slave-4cx3 krb5kdc[7808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713165, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:19:25.929512 8782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:25.911466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:52467 (local address 127.27.185.126:35879)
0929 20:19:25.911959 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:25.911964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:25.912920 (+ 956us) server_negotiation.cc:407] Connection header received
0929 20:19:25.914705 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:25.914711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:25.914829 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:25.914954 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:25.916479 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:25.917196 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:25.918925 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:25.919466 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:25.919605 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:25.923351 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:25.923356 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:25.923359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:25.923463 (+ 104us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:25.926475 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:25.927224 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:25.927228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:25.927230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:25.927372 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:25.927904 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:25.927909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:25.927911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:25.928216 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:25.928345 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:25.929029 (+ 684us) server_negotiation.cc:299] Negotiation successful
0929 20:19:25.929201 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":270,"threads_started":1}
I20260929 20:19:25.930501 8783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:25.912020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:35879 (local address 127.27.185.67:52467)
0929 20:19:25.912723 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:25.912768 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:25.914423 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:25.915201 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:25.915210 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:25.915701 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:19:25.916303 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:25.916322 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:25.917362 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:25.917367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:25.918764 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:25.918775 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:25.919714 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:25.919721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:25.920051 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:25.920851 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:19:25.920875 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:19:25.923143 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:19:25.926630 (+ 3487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:25.926637 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:25.926652 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:25.927105 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:25.927555 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:25.927559 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:25.927561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:25.927766 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:25.928337 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:19:25.928343 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:19:25.928691 (+ 348us) client_negotiation.cc:769] Sending connection context
0929 20:19:25.928926 (+ 235us) client_negotiation.cc:240] Negotiation successful
0929 20:19:25.929335 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":281,"threads_started":1}
I20260929 20:19:25.932058 8780 heartbeater.cc:347] Connected to a master server at 127.27.185.126:35879
I20260929 20:19:25.932387 8780 heartbeater.cc:464] Registering TS with master...
I20260929 20:19:25.933121 8780 heartbeater.cc:511] Master 127.27.185.126:35879 requested a full tablet report, sending...
I20260929 20:19:25.934821 8283 ts_manager.cc:194] Registered new tserver with Master: 3bef4fa96cd24026bc828f3b6d324eba (127.27.185.67:45287)
I20260929 20:19:25.935822 8283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.67@KRBTEST.COM'} at 127.27.185.67:52467
I20260929 20:19:25.947685 28389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:19:25.949501 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 8364
Sep 29 20:19:25 dist-test-slave-4cx3 krb5kdc[7808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.65@KRBTEST.COM
I20260929 20:19:25.966624 8785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:25.950447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60966 (local address 127.27.185.65:32983)
0929 20:19:25.951112 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:25.951122 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:25.951153 (+ 31us) server_negotiation.cc:407] Connection header received
0929 20:19:25.951265 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:25.951281 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:25.951643 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:25.951821 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:25.953558 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:25.954680 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:25.956096 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:25.956671 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:25.956823 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:25.961218 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:25.961231 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:25.961247 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:25.961285 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:25.964157 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:25.964828 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:25.964835 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:25.964837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:25.964981 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:25.965310 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:25.965313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:25.965315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:25.965641 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:25.965870 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:25.966132 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 20:19:25.966329 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":305,"threads_started":1}
I20260929 20:19:25.968034 8498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:19:26.052029 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 8364
I20260929 20:19:26.068224 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 8509
Sep 29 20:19:26 dist-test-slave-4cx3 krb5kdc[7808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.66@KRBTEST.COM
I20260929 20:19:26.085390 8787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:26.069098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34500 (local address 127.27.185.66:35725)
0929 20:19:26.069684 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:26.069692 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:26.069718 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:19:26.069791 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:26.069798 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:26.070016 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:26.070243 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:26.071528 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:26.072612 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:26.074000 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:26.074580 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:26.074703 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:26.079217 (+ 4514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:26.079230 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:26.079245 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:26.079281 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:26.082311 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:26.083023 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:26.083032 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:26.083035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:26.083184 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:26.083817 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:26.083823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:26.083826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:26.084226 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:26.084480 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:26.084748 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 20:19:26.084980 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":334,"threads_started":1}
I20260929 20:19:26.086858 8641 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:19:26.168478 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 8509
I20260929 20:19:26.184257 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 8652
Sep 29 20:19:26 dist-test-slave-4cx3 krb5kdc[7808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.67@KRBTEST.COM
I20260929 20:19:26.200794 8789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:26.185276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59114 (local address 127.27.185.67:45287)
0929 20:19:26.185882 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:26.185888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:26.185917 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:19:26.185986 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:26.185993 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:26.186209 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:26.186380 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:26.187814 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:26.188843 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:26.190329 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:26.190896 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:26.191057 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:26.195629 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:26.195641 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:26.195658 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:26.195692 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:26.198345 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:26.199055 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:26.199061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:26.199063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:26.199179 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:26.199541 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:26.199544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:26.199546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:26.199876 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:26.200117 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:26.200322 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 20:19:26.200507 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":312,"threads_started":1}
I20260929 20:19:26.202111 8775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:19:26.287886 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 8652
I20260929 20:19:26.304116 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 8252
I20260929 20:19:26.305573 8314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:19:26.412163 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 8252
I20260929 20:19:26.434489 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:35879
--webserver_interface=127.27.185.126
--webserver_port=45281
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:37481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:35879
--ranger_config_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.185.84:45025/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.185.84:45025/kms with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:19:26.780977 8792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:19:26.781257 8792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:19:26.781453 8792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:19:26.789070 8792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:19:26.789175 8792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:19:26.789268 8792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:19:26.789391 8792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:19:26.789479 8792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:19:26.799729 8792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:37481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.185.84:45025/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:35879
--ranger_config_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:35879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=45281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:19:26.801070 8792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:19:26.802582 8792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:19:26.816273 8798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:26.816427 8797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:26.816666 8800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:19:26.817128 8792 server_base.cc:1034] running on GCE node
I20260929 20:19:26.817884 8792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:19:26.819334 8792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:19:26.820600 8792 hybrid_clock.cc:648] HybridClock initialized: now 1790713166820560 us; error 38 us; skew 500 ppm
I20260929 20:19:26.820961 8792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:19:26 dist-test-slave-4cx3 krb5kdc[7808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713166, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:19:26.824055 8792 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:19:26.825569 8792 webserver.cc:460] Webserver started at http://127.27.185.126:45281/ using document root <none> and password file <none>
I20260929 20:19:26.826174 8792 fs_manager.cc:359] Metadata directory not provided
I20260929 20:19:26.826279 8792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 20:19:26 dist-test-slave-4cx3 krb5kdc[7808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713166, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for HTTP/127.27.185.84@KRBTEST.COM
I20260929 20:19:26.891355 8792 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 20:19:26.894784 8808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:26.896345 8792 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 20:19:26.896574 8792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "3d123993bcaf4172a8045f5e988582b6"
format_stamp: "Formatted at 2026-09-29 20:19:22 on dist-test-slave-4cx3"
server_key: "ed97eea3673fb12e5d7aebf376d5c20e"
server_key_iv: "88075ac467e8abf6e355617f1906a24c"
server_key_version: "kuduclusterkey@0"
I20260929 20:19:26.897626 8792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:19:26.910599 8792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:19:26.914125 8792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:19:26.914409 8792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:19:26.932888 8792 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:35879
I20260929 20:19:26.932919 8859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:35879 every 8 connection(s)
I20260929 20:19:26.934494 8792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:19:26.938410 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 8792
I20260929 20:19:26.939258 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.185.84:45025/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:32983
--local_ip_for_outbound_sockets=127.27.185.65
--tserver_master_addrs=127.27.185.126:35879
--webserver_port=42445
--webserver_interface=127.27.185.65
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:37481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.185.84:45025/kms with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:19:26.941359 8860 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:19:26.944687 8860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6: Bootstrap starting.
I20260929 20:19:26.963483 8860 log.cc:824] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6: Log is configured to *not* fsync() on all Append() calls
I20260929 20:19:26.972584 8860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:19:26.973136 8860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6: Bootstrap complete.
I20260929 20:19:26.982328 8860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d123993bcaf4172a8045f5e988582b6" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 35879 } }
I20260929 20:19:26.983222 8860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d123993bcaf4172a8045f5e988582b6, State: Initialized, Role: FOLLOWER
I20260929 20:19:26.983935 8860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d123993bcaf4172a8045f5e988582b6" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 35879 } }
I20260929 20:19:26.984237 8860 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:19:26.984366 8860 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:19:26.984560 8860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [term 1 FOLLOWER]: Advancing to term 2
I20260929 20:19:26.988644 8860 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d123993bcaf4172a8045f5e988582b6" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 35879 } }
I20260929 20:19:26.989162 8860 leader_election.cc:302] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d123993bcaf4172a8045f5e988582b6; no voters:
I20260929 20:19:26.989857 8860 leader_election.cc:288] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 20:19:26.990025 8864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 20:19:26.991114 8864 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [term 2 LEADER]: Becoming Leader. State: Replica: 3d123993bcaf4172a8045f5e988582b6, State: Running, Role: LEADER
I20260929 20:19:26.991804 8864 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d123993bcaf4172a8045f5e988582b6" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 35879 } }
I20260929 20:19:26.992192 8860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:19:26.994611 8866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d123993bcaf4172a8045f5e988582b6. Latest consensus state: current_term: 2 leader_uuid: "3d123993bcaf4172a8045f5e988582b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d123993bcaf4172a8045f5e988582b6" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 35879 } } }
I20260929 20:19:26.994971 8866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [sys.catalog]: This master's current role is: LEADER
I20260929 20:19:26.994879 8865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3d123993bcaf4172a8045f5e988582b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d123993bcaf4172a8045f5e988582b6" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 35879 } } }
I20260929 20:19:26.995285 8865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6 [sys.catalog]: This master's current role is: LEADER
I20260929 20:19:27.004925 8871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:19:27.011231 8871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:19:27.012127 8860 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 20:19:27.012810 8871 catalog_manager.cc:1259] Loaded cluster ID: 68aeddefdcbd41d58dc8e11598ef9c8c
I20260929 20:19:27.012984 8871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:19:27.017127 8871 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:19:27.018661 8871 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6: Loaded TSK: 0
I20260929 20:19:27.020272 8871 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 20:19:27.354856 8862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:19:27.355320 8862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:19:27.355579 8862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:19:27.365722 8862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:19:27.365969 8862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:19:27.366214 8862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:19:27.382156 8862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:37481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.185.84:45025/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:32983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=42445
--tserver_master_addrs=127.27.185.126:35879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:19:27.384371 8862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:19:27.386196 8862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:19:27.418254 8898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:27.419482 8899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:19:27.424646 8862 server_base.cc:1034] running on GCE node
W20260929 20:19:27.425058 8901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:19:27.425923 8862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:19:27.432088 8862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:19:27.435693 8862 hybrid_clock.cc:648] HybridClock initialized: now 1790713167435511 us; error 177 us; skew 500 ppm
I20260929 20:19:27.436239 8862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:19:27 dist-test-slave-4cx3 krb5kdc[7808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713167, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:19:27.440745 8862 init.cc:377] Logged in from keytab as kudu/127.27.185.65@KRBTEST.COM (short username kudu)
I20260929 20:19:27.457223 8862 webserver.cc:460] Webserver started at http://127.27.185.65:42445/ using document root <none> and password file <none>
I20260929 20:19:27.458065 8862 fs_manager.cc:359] Metadata directory not provided
I20260929 20:19:27.458242 8862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 20:19:27 dist-test-slave-4cx3 krb5kdc[7808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713167, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for HTTP/127.27.185.84@KRBTEST.COM
I20260929 20:19:27.483791 8862 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 20:19:27.494060 8914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:27.496224 8862 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.007s
I20260929 20:19:27.496490 8862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "9a70c4b2e1764708a9dd9c07c329f9a3"
format_stamp: "Formatted at 2026-09-29 20:19:24 on dist-test-slave-4cx3"
server_key: "d23cc0db5f88a5787435718ed237a655"
server_key_iv: "0f0860acc672126dc069d683d3e5c20f"
server_key_version: "kuduclusterkey@0"
I20260929 20:19:27.498082 8862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:19:27.557493 8862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:19:27.565819 8862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:19:27.566256 8862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:19:27.570158 8862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:19:27.572613 8862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:19:27.572738 8862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:27.572860 8862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:19:27.572942 8862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:27.648748 8862 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:32983
I20260929 20:19:27.648806 9027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:32983 every 8 connection(s)
I20260929 20:19:27.650851 8862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:19:27.658140 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 8862
I20260929 20:19:27.659224 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.185.84:45025/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:35725
--local_ip_for_outbound_sockets=127.27.185.66
--tserver_master_addrs=127.27.185.126:35879
--webserver_port=34441
--webserver_interface=127.27.185.66
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:37481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.185.84:45025/kms with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:19:27 dist-test-slave-4cx3 krb5kdc[7808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713167, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:19:27.701323 9031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:27.654594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:35879 (local address 127.27.185.65:43205)
0929 20:19:27.666398 (+ 11804us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:27.666458 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:27.670676 (+ 4218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:27.673320 (+ 2644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:27.673333 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:27.674014 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:19:27.674839 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:27.674867 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:27.677441 (+ 2574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:27.677449 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:27.679738 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:27.679768 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:27.681834 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:27.681843 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:27.682222 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:27.686976 (+ 4754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:19:27.687014 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:19:27.690720 (+ 3706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:19:27.695528 (+ 4808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:27.695538 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:27.695558 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:27.696242 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:27.696874 (+ 632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:27.696881 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:27.696885 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:27.697490 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:27.698509 (+ 1019us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:19:27.698518 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:19:27.698939 (+ 421us) client_negotiation.cc:769] Sending connection context
0929 20:19:27.699233 (+ 294us) client_negotiation.cc:240] Negotiation successful
0929 20:19:27.699874 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11442,"thread_start_us":12700,"threads_started":1}
I20260929 20:19:27.701337 9030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:27.654502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:43205 (local address 127.27.185.126:35879)
0929 20:19:27.665711 (+ 11209us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:27.665739 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:27.668332 (+ 2593us) server_negotiation.cc:407] Connection header received
0929 20:19:27.671524 (+ 3192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:27.671551 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:27.672094 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:27.672509 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:27.675085 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:27.677224 (+ 2139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:27.679928 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:27.681143 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:27.681330 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:27.691145 (+ 9815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:27.691158 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:27.691181 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:27.691243 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:27.695321 (+ 4078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:27.696446 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:27.696459 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:27.696472 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:27.696701 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:27.697626 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:27.697632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:27.697636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:27.698336 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:27.698524 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:27.699576 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0929 20:19:27.700018 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10850,"thread_start_us":10604,"threads_started":1}
I20260929 20:19:27.705790 9028 heartbeater.cc:347] Connected to a master server at 127.27.185.126:35879
I20260929 20:19:27.706118 9028 heartbeater.cc:464] Registering TS with master...
I20260929 20:19:27.706975 9028 heartbeater.cc:511] Master 127.27.185.126:35879 requested a full tablet report, sending...
W20260929 20:19:27.710323 9028 heartbeater.cc:644] Failed to heartbeat to 127.27.185.126:35879 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 20:19:27.710542 9028 heartbeater.cc:464] Registering TS with master...
I20260929 20:19:27.710809 9028 heartbeater.cc:511] Master 127.27.185.126:35879 requested a full tablet report, sending...
I20260929 20:19:27.712085 9028 heartbeater.cc:464] Registering TS with master...
I20260929 20:19:27.712338 9028 heartbeater.cc:511] Master 127.27.185.126:35879 requested a full tablet report, sending...
W20260929 20:19:27.713469 9028 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 20:19:28.069363 9032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:19:28.069972 9032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:19:28.070170 9032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:19:28.078148 9032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:19:28.078311 9032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:19:28.078508 9032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:19:28.089135 9032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:37481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.185.84:45025/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:35725
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=34441
--tserver_master_addrs=127.27.185.126:35879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:19:28.090585 9032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:19:28.092034 9032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:19:28.108053 9044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:28.108090 9045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:28.109754 9047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:19:28.110145 9032 server_base.cc:1034] running on GCE node
I20260929 20:19:28.110924 9032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:19:28.112205 9032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:19:28.113442 9032 hybrid_clock.cc:648] HybridClock initialized: now 1790713168113379 us; error 58 us; skew 500 ppm
I20260929 20:19:28.113772 9032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:19:28 dist-test-slave-4cx3 krb5kdc[7808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713168, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:19:28.117165 9032 init.cc:377] Logged in from keytab as kudu/127.27.185.66@KRBTEST.COM (short username kudu)
I20260929 20:19:28.122756 9032 webserver.cc:460] Webserver started at http://127.27.185.66:34441/ using document root <none> and password file <none>
I20260929 20:19:28.123687 9032 fs_manager.cc:359] Metadata directory not provided
I20260929 20:19:28.123852 9032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 20:19:28 dist-test-slave-4cx3 krb5kdc[7808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713168, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for HTTP/127.27.185.84@KRBTEST.COM
I20260929 20:19:28.143676 9032 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 20:19:28.147636 9055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:28.149262 9032 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:19:28.149469 9032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "a588c33cea4a4a939a3c6ee7b0f85bb3"
format_stamp: "Formatted at 2026-09-29 20:19:25 on dist-test-slave-4cx3"
server_key: "e84bcba5dbfff4b962469f56cd08bd05"
server_key_iv: "c06918f558e7f2256b9ed1c832f9ba4d"
server_key_version: "kuduclusterkey@0"
I20260929 20:19:28.150511 9032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:19:28.184183 9032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:19:28.189248 9032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:19:28.189566 9032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:19:28.190754 9032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:19:28.192319 9032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:19:28.192446 9032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:28.192613 9032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:19:28.192754 9032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 20:19:28.275676 9032 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:35725
I20260929 20:19:28.275738 9171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:35725 every 8 connection(s)
I20260929 20:19:28.277213 9032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:19:28.278542 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 9032
I20260929 20:19:28.279295 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.185.84:45025/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:45287
--local_ip_for_outbound_sockets=127.27.185.67
--tserver_master_addrs=127.27.185.126:35879
--webserver_port=45985
--webserver_interface=127.27.185.67
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:37481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.185.84:45025/kms with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:19:28 dist-test-slave-4cx3 krb5kdc[7808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713168, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:19:28.322700 9175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:28.281495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:35879 (local address 127.27.185.66:38489)
0929 20:19:28.299173 (+ 17678us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:28.299216 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:28.302126 (+ 2910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:28.303027 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:28.303036 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:28.303645 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:19:28.304506 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:28.304532 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:28.306185 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:28.306191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:28.308254 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:28.308279 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:28.309613 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:28.309620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:28.309903 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:28.310720 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:19:28.310752 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:19:28.313609 (+ 2857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:19:28.317745 (+ 4136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:28.317754 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:28.317773 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:28.318387 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:28.318887 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:28.318891 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:28.318894 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:28.319142 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:28.319907 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:19:28.319915 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:19:28.320377 (+ 462us) client_negotiation.cc:769] Sending connection context
0929 20:19:28.320704 (+ 327us) client_negotiation.cc:240] Negotiation successful
0929 20:19:28.321281 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17394,"thread_start_us":17776,"threads_started":1}
I20260929 20:19:28.323832 9174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:28.280692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:38489 (local address 127.27.185.126:35879)
0929 20:19:28.298767 (+ 18075us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:28.298776 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:28.299372 (+ 596us) server_negotiation.cc:407] Connection header received
0929 20:19:28.302505 (+ 3133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:28.302511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:28.302605 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:28.302737 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:28.304742 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:28.305976 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:28.308461 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:28.309130 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:28.309253 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:28.313883 (+ 4630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:28.313891 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:28.313896 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:28.313938 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:28.317573 (+ 3635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:28.318589 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:28.318595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:28.318599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:28.318751 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:28.319281 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:28.319285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:28.319289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:28.319688 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:28.319835 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:28.323232 (+ 3397us) server_negotiation.cc:299] Negotiation successful
0929 20:19:28.323494 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17876,"thread_start_us":18088,"threads_started":1}
I20260929 20:19:28.326088 9172 heartbeater.cc:347] Connected to a master server at 127.27.185.126:35879
I20260929 20:19:28.326411 9172 heartbeater.cc:464] Registering TS with master...
I20260929 20:19:28.327311 9172 heartbeater.cc:511] Master 127.27.185.126:35879 requested a full tablet report, sending...
I20260929 20:19:28.329533 8825 ts_manager.cc:194] Registered new tserver with Master: a588c33cea4a4a939a3c6ee7b0f85bb3 (127.27.185.66:35725)
I20260929 20:19:28.331821 8825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.66@KRBTEST.COM'} at 127.27.185.66:38489
Sep 29 20:19:28 dist-test-slave-4cx3 krb5kdc[7808](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790713168, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:19:28.713616 9028 heartbeater.cc:464] Registering TS with master...
I20260929 20:19:28.714066 9028 heartbeater.cc:511] Master 127.27.185.126:35879 requested a full tablet report, sending...
I20260929 20:19:28.715406 8825 ts_manager.cc:194] Registered new tserver with Master: 9a70c4b2e1764708a9dd9c07c329f9a3 (127.27.185.65:32983)
I20260929 20:19:28.716614 8825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.65@KRBTEST.COM'} at 127.27.185.65:43205
W20260929 20:19:28.744660 9176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:19:28.745098 9176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:19:28.745270 9176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:19:28.757261 9176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:19:28.757434 9176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:19:28.757647 9176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:19:28.773285 9176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:37481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.185.84:45025/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:45287
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=45985
--tserver_master_addrs=127.27.185.126:35879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:19:28.775038 9176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:19:28.776862 9176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:19:28.795198 9183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:28.796983 9182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:19:28.799649 9176 server_base.cc:1034] running on GCE node
W20260929 20:19:28.800001 9185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:19:28.800832 9176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:19:28.802207 9176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:19:28.803471 9176 hybrid_clock.cc:648] HybridClock initialized: now 1790713168803391 us; error 69 us; skew 500 ppm
I20260929 20:19:28.803946 9176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:19:28 dist-test-slave-4cx3 krb5kdc[7808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713168, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:19:28.808514 9176 init.cc:377] Logged in from keytab as kudu/127.27.185.67@KRBTEST.COM (short username kudu)
I20260929 20:19:28.813120 9176 webserver.cc:460] Webserver started at http://127.27.185.67:45985/ using document root <none> and password file <none>
I20260929 20:19:28.814111 9176 fs_manager.cc:359] Metadata directory not provided
I20260929 20:19:28.814280 9176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 20:19:28 dist-test-slave-4cx3 krb5kdc[7808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713168, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for HTTP/127.27.185.84@KRBTEST.COM
I20260929 20:19:28.847834 9176 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.006s sys 0.001s
I20260929 20:19:28.851970 9193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:28.853901 9176 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 20:19:28.854104 9176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "3bef4fa96cd24026bc828f3b6d324eba"
format_stamp: "Formatted at 2026-09-29 20:19:25 on dist-test-slave-4cx3"
server_key: "65b9457741a967eaf81de9ad961dc7db"
server_key_iv: "9cf8e45d77be1d3b3db35bde2a7a75de"
server_key_version: "kuduclusterkey@0"
I20260929 20:19:28.855563 9176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 29 20:19:28 dist-test-slave-4cx3 krb5kdc[7808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713168, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for HTTP/127.27.185.84@KRBTEST.COM
I20260929 20:19:28.879034 9176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:19:28.884356 9176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:19:28.884721 9176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:19:28.888684 9176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:19:28.897431 9176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:19:28.897580 9176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:28.897742 9176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:19:28.897840 9176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:29.072031 9176 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:45287
I20260929 20:19:29.072095 9305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:45287 every 8 connection(s)
I20260929 20:19:29.073980 9176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:19:29.075426 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 9176
I20260929 20:19:29.108987 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:29.076964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43548 (local address 127.27.185.126:35879)
0929 20:19:29.084656 (+ 7692us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:29.084662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:29.084684 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:19:29.084755 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:29.084762 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:29.084846 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:29.084972 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:29.087270 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.088841 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.095175 (+ 6334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.096196 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.096417 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:29.099948 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:29.099960 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:29.099968 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:29.100054 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:29.105133 (+ 5079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:29.106096 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:29.106106 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:29.106113 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:29.106333 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:29.106797 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:29.106805 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:29.106811 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:29.107340 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:29.107700 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:29.108058 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 20:19:29.108398 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7562,"thread_start_us":7431,"threads_started":1}
Sep 29 20:19:29 dist-test-slave-4cx3 krb5kdc[7808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713168, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:19:29.131222 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:29.087002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:50095 (local address 127.27.185.126:35879)
0929 20:19:29.110464 (+ 23462us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:29.110474 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:29.112844 (+ 2370us) server_negotiation.cc:407] Connection header received
0929 20:19:29.114338 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:29.114344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:29.114452 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:29.114570 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:29.116734 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.117622 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.119556 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.120018 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.120160 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:29.124235 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:29.124241 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:29.124246 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:29.124291 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:29.127750 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:29.128572 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:29.128578 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:29.128582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:29.128721 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:29.129229 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:29.129235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:29.129238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:29.129562 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:29.129735 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:29.130542 (+ 807us) server_negotiation.cc:299] Negotiation successful
0929 20:19:29.130819 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23274,"thread_start_us":28674,"threads_started":1}
I20260929 20:19:29.132478 9310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:29.086298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:35879 (local address 127.27.185.67:50095)
0929 20:19:29.111650 (+ 25352us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:29.111731 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:29.113961 (+ 2230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:29.115075 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:29.115088 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:29.115870 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:19:29.116544 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:29.116563 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.117821 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.117826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:29.119333 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:29.119356 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.120276 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.120283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:29.120557 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:29.121345 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:19:29.121371 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:19:29.123971 (+ 2600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:19:29.127914 (+ 3943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:29.127923 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:29.127951 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:29.128427 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:29.128841 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:29.128849 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:29.128854 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:29.129091 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:29.129705 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:19:29.129714 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:19:29.130125 (+ 411us) client_negotiation.cc:769] Sending connection context
0929 20:19:29.130430 (+ 305us) client_negotiation.cc:240] Negotiation successful
0929 20:19:29.131064 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24917,"thread_start_us":28277,"threads_started":1}
I20260929 20:19:29.134217 9306 heartbeater.cc:347] Connected to a master server at 127.27.185.126:35879
I20260929 20:19:29.134578 9306 heartbeater.cc:464] Registering TS with master...
I20260929 20:19:29.135454 9306 heartbeater.cc:511] Master 127.27.185.126:35879 requested a full tablet report, sending...
I20260929 20:19:29.137816 8825 ts_manager.cc:194] Registered new tserver with Master: 3bef4fa96cd24026bc828f3b6d324eba (127.27.185.67:45287)
I20260929 20:19:29.138793 8825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.67@KRBTEST.COM'} at 127.27.185.67:50095
I20260929 20:19:29.141644 28389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:19:29.175202 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:29.154291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43554 (local address 127.27.185.126:35879)
0929 20:19:29.154567 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:29.154575 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:29.155183 (+ 608us) server_negotiation.cc:407] Connection header received
0929 20:19:29.155413 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:29.155419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:29.155530 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:29.155658 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:29.158160 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.159326 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.161779 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.162459 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.162603 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:29.166498 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:29.166505 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:29.166510 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:29.166561 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:29.170451 (+ 3890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:29.171519 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:29.171525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:29.171529 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:29.171698 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:29.172286 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:29.172292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:29.172296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:29.172712 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:29.173099 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:29.174524 (+ 1425us) server_negotiation.cc:299] Negotiation successful
0929 20:19:29.174780 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 20:19:29.181852 8824 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:19:29.193596 8824 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 29 20:19:29 dist-test-slave-4cx3 krb5kdc[7808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713168, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for HTTP/127.27.185.84@KRBTEST.COM
I20260929 20:19:29.258209 9322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:29.214789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:32983 (local address 127.0.0.1:60978)
0929 20:19:29.238954 (+ 24165us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:29.239026 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:29.239258 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:29.242058 (+ 2800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:29.242070 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:29.242123 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:19:29.242662 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:29.242684 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.245176 (+ 2492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.245184 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:29.253015 (+ 7831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:29.253050 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.256985 (+ 3935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.256994 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:29.257261 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:29.257333 (+ 72us) client_negotiation.cc:769] Sending connection context
0929 20:19:29.257520 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 20:19:29.257770 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23866,"thread_start_us":23727,"threads_started":1}
I20260929 20:19:29.261337 9324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:29.215435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60978 (local address 127.27.185.65:32983)
0929 20:19:29.241382 (+ 25947us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:29.241392 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:29.241420 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:19:29.241513 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:29.241520 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:29.241746 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:29.241941 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:19:29.242966 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.244967 (+ 2001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.254159 (+ 9192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.256300 (+ 2141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.256476 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:29.256768 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:29.260637 (+ 3869us) server_negotiation.cc:299] Negotiation successful
0929 20:19:29.260865 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25777,"thread_start_us":25623,"threads_started":1}
I20260929 20:19:29.266068 9325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:29.215480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34508 (local address 127.27.185.66:35725)
0929 20:19:29.238112 (+ 22632us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:29.238123 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:29.240106 (+ 1983us) server_negotiation.cc:407] Connection header received
0929 20:19:29.240224 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:29.240233 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:29.240509 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:29.240706 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:19:29.257850 (+ 17144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.259457 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.262854 (+ 3397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.264501 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.264657 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:29.264877 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:29.265526 (+ 649us) server_negotiation.cc:299] Negotiation successful
0929 20:19:29.265712 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22400,"thread_start_us":22661,"threads_started":1}
I20260929 20:19:29.266930 9323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:29.214895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:35725 (local address 127.0.0.1:34508)
0929 20:19:29.239828 (+ 24933us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:29.239845 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:29.239964 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:29.242211 (+ 2247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:29.242217 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:29.242246 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:19:29.257597 (+ 15351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:29.257620 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.260582 (+ 2962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.260588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:29.262649 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:29.262669 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.265063 (+ 2394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.265071 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:29.265263 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:29.265299 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 20:19:29.266392 (+ 1093us) client_negotiation.cc:240] Negotiation successful
0929 20:19:29.266551 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24798,"thread_start_us":27579,"threads_started":1}
I20260929 20:19:29.267242 8962 tablet_service.cc:1467] Processing CreateTablet for tablet 3585b23f63d54468989f6b915fc8ed7f (DEFAULT_TABLE table=test-table [id=2c1c10a184f04e12a0cec2a9c8a24acf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:19:29.269338 8962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3585b23f63d54468989f6b915fc8ed7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:19:29.276212 9322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:29.252728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:45287 (local address 127.0.0.1:59120)
0929 20:19:29.260249 (+ 7521us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:29.260274 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:29.260424 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:29.268839 (+ 8415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:29.268844 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:29.268874 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:19:29.269056 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:29.269069 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.271431 (+ 2362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.271437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:29.273334 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:29.273361 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.275442 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.275449 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:29.275633 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:29.275672 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 20:19:29.275748 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 20:19:29.275901 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6527,"thread_start_us":13462,"threads_started":1}
I20260929 20:19:29.277299 9326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:29.243284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59120 (local address 127.27.185.67:45287)
0929 20:19:29.267948 (+ 24664us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:29.267958 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:29.267995 (+ 37us) server_negotiation.cc:407] Connection header received
0929 20:19:29.268098 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:29.268106 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:29.268376 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:29.269246 (+ 870us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:19:29.269571 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.271253 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.273588 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.275225 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.276480 (+ 1255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:29.276713 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:29.276792 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 20:19:29.276965 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24467,"thread_start_us":25259,"threads_started":1}
I20260929 20:19:29.282330 9241 tablet_service.cc:1467] Processing CreateTablet for tablet 3585b23f63d54468989f6b915fc8ed7f (DEFAULT_TABLE table=test-table [id=2c1c10a184f04e12a0cec2a9c8a24acf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:19:29.284312 9241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3585b23f63d54468989f6b915fc8ed7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:19:29.295854 9105 tablet_service.cc:1467] Processing CreateTablet for tablet 3585b23f63d54468989f6b915fc8ed7f (DEFAULT_TABLE table=test-table [id=2c1c10a184f04e12a0cec2a9c8a24acf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:19:29.297698 9105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3585b23f63d54468989f6b915fc8ed7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:19:29.311570 9328 tablet_bootstrap.cc:492] T 3585b23f63d54468989f6b915fc8ed7f P 3bef4fa96cd24026bc828f3b6d324eba: Bootstrap starting.
I20260929 20:19:29.315433 9328 tablet_bootstrap.cc:654] T 3585b23f63d54468989f6b915fc8ed7f P 3bef4fa96cd24026bc828f3b6d324eba: Neither blocks nor log segments found. Creating new log.
I20260929 20:19:29.318723 9328 log.cc:824] T 3585b23f63d54468989f6b915fc8ed7f P 3bef4fa96cd24026bc828f3b6d324eba: Log is configured to *not* fsync() on all Append() calls
I20260929 20:19:29.326233 9328 tablet_bootstrap.cc:492] T 3585b23f63d54468989f6b915fc8ed7f P 3bef4fa96cd24026bc828f3b6d324eba: No bootstrap required, opened a new log
I20260929 20:19:29.326570 9328 ts_tablet_manager.cc:1397] T 3585b23f63d54468989f6b915fc8ed7f P 3bef4fa96cd24026bc828f3b6d324eba: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20260929 20:19:29.329959 9331 tablet_bootstrap.cc:492] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3: Bootstrap starting.
I20260929 20:19:29.336664 9331 tablet_bootstrap.cc:654] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3: Neither blocks nor log segments found. Creating new log.
I20260929 20:19:29.336797 9329 tablet_bootstrap.cc:492] T 3585b23f63d54468989f6b915fc8ed7f P 9a70c4b2e1764708a9dd9c07c329f9a3: Bootstrap starting.
I20260929 20:19:29.336755 9172 heartbeater.cc:503] Master 127.27.185.126:35879 was elected leader, sending a full tablet report...
I20260929 20:19:29.341126 9329 tablet_bootstrap.cc:654] T 3585b23f63d54468989f6b915fc8ed7f P 9a70c4b2e1764708a9dd9c07c329f9a3: Neither blocks nor log segments found. Creating new log.
I20260929 20:19:29.341704 9331 log.cc:824] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3: Log is configured to *not* fsync() on all Append() calls
I20260929 20:19:29.346204 9328 raft_consensus.cc:348] T 3585b23f63d54468989f6b915fc8ed7f P 3bef4fa96cd24026bc828f3b6d324eba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a70c4b2e1764708a9dd9c07c329f9a3" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 32983 } } peers { permanent_uuid: "a588c33cea4a4a939a3c6ee7b0f85bb3" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 35725 } } peers { permanent_uuid: "3bef4fa96cd24026bc828f3b6d324eba" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45287 } }
I20260929 20:19:29.346737 9328 raft_consensus.cc:374] T 3585b23f63d54468989f6b915fc8ed7f P 3bef4fa96cd24026bc828f3b6d324eba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:19:29.346858 9328 raft_consensus.cc:729] T 3585b23f63d54468989f6b915fc8ed7f P 3bef4fa96cd24026bc828f3b6d324eba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bef4fa96cd24026bc828f3b6d324eba, State: Initialized, Role: FOLLOWER
I20260929 20:19:29.347720 9328 consensus_queue.cc:260] T 3585b23f63d54468989f6b915fc8ed7f P 3bef4fa96cd24026bc828f3b6d324eba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a70c4b2e1764708a9dd9c07c329f9a3" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 32983 } } peers { permanent_uuid: "a588c33cea4a4a939a3c6ee7b0f85bb3" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 35725 } } peers { permanent_uuid: "3bef4fa96cd24026bc828f3b6d324eba" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45287 } }
I20260929 20:19:29.352422 9331 tablet_bootstrap.cc:492] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3: No bootstrap required, opened a new log
I20260929 20:19:29.352859 9331 ts_tablet_manager.cc:1397] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.008s
I20260929 20:19:29.359876 9329 log.cc:824] T 3585b23f63d54468989f6b915fc8ed7f P 9a70c4b2e1764708a9dd9c07c329f9a3: Log is configured to *not* fsync() on all Append() calls
I20260929 20:19:29.360899 9306 heartbeater.cc:503] Master 127.27.185.126:35879 was elected leader, sending a full tablet report...
I20260929 20:19:29.361898 9328 ts_tablet_manager.cc:1428] T 3585b23f63d54468989f6b915fc8ed7f P 3bef4fa96cd24026bc828f3b6d324eba: Time spent starting tablet: real 0.035s user 0.016s sys 0.011s
I20260929 20:19:29.368142 9331 raft_consensus.cc:348] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a70c4b2e1764708a9dd9c07c329f9a3" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 32983 } } peers { permanent_uuid: "a588c33cea4a4a939a3c6ee7b0f85bb3" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 35725 } } peers { permanent_uuid: "3bef4fa96cd24026bc828f3b6d324eba" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45287 } }
I20260929 20:19:29.368814 9331 raft_consensus.cc:374] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:19:29.368978 9331 raft_consensus.cc:729] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a588c33cea4a4a939a3c6ee7b0f85bb3, State: Initialized, Role: FOLLOWER
I20260929 20:19:29.369866 9331 consensus_queue.cc:260] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a70c4b2e1764708a9dd9c07c329f9a3" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 32983 } } peers { permanent_uuid: "a588c33cea4a4a939a3c6ee7b0f85bb3" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 35725 } } peers { permanent_uuid: "3bef4fa96cd24026bc828f3b6d324eba" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45287 } }
I20260929 20:19:29.378418 9331 ts_tablet_manager.cc:1428] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3: Time spent starting tablet: real 0.025s user 0.014s sys 0.006s
I20260929 20:19:29.384891 9329 tablet_bootstrap.cc:492] T 3585b23f63d54468989f6b915fc8ed7f P 9a70c4b2e1764708a9dd9c07c329f9a3: No bootstrap required, opened a new log
I20260929 20:19:29.385299 9329 ts_tablet_manager.cc:1397] T 3585b23f63d54468989f6b915fc8ed7f P 9a70c4b2e1764708a9dd9c07c329f9a3: Time spent bootstrapping tablet: real 0.051s user 0.010s sys 0.006s
I20260929 20:19:29.397591 9329 raft_consensus.cc:348] T 3585b23f63d54468989f6b915fc8ed7f P 9a70c4b2e1764708a9dd9c07c329f9a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a70c4b2e1764708a9dd9c07c329f9a3" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 32983 } } peers { permanent_uuid: "a588c33cea4a4a939a3c6ee7b0f85bb3" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 35725 } } peers { permanent_uuid: "3bef4fa96cd24026bc828f3b6d324eba" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45287 } }
I20260929 20:19:29.398195 9329 raft_consensus.cc:374] T 3585b23f63d54468989f6b915fc8ed7f P 9a70c4b2e1764708a9dd9c07c329f9a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:19:29.398315 9329 raft_consensus.cc:729] T 3585b23f63d54468989f6b915fc8ed7f P 9a70c4b2e1764708a9dd9c07c329f9a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a70c4b2e1764708a9dd9c07c329f9a3, State: Initialized, Role: FOLLOWER
I20260929 20:19:29.399235 9329 consensus_queue.cc:260] T 3585b23f63d54468989f6b915fc8ed7f P 9a70c4b2e1764708a9dd9c07c329f9a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a70c4b2e1764708a9dd9c07c329f9a3" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 32983 } } peers { permanent_uuid: "a588c33cea4a4a939a3c6ee7b0f85bb3" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 35725 } } peers { permanent_uuid: "3bef4fa96cd24026bc828f3b6d324eba" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45287 } }
I20260929 20:19:29.418499 9329 ts_tablet_manager.cc:1428] T 3585b23f63d54468989f6b915fc8ed7f P 9a70c4b2e1764708a9dd9c07c329f9a3: Time spent starting tablet: real 0.033s user 0.029s sys 0.005s
I20260929 20:19:29.419092 9028 heartbeater.cc:503] Master 127.27.185.126:35879 was elected leader, sending a full tablet report...
W20260929 20:19:29.548503 9173 tablet.cc:2369] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:19:29.588678 9307 tablet.cc:2369] T 3585b23f63d54468989f6b915fc8ed7f P 3bef4fa96cd24026bc828f3b6d324eba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:19:29.642349 9335 raft_consensus.cc:482] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:19:29.642693 9335 raft_consensus.cc:504] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a70c4b2e1764708a9dd9c07c329f9a3" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 32983 } } peers { permanent_uuid: "a588c33cea4a4a939a3c6ee7b0f85bb3" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 35725 } } peers { permanent_uuid: "3bef4fa96cd24026bc828f3b6d324eba" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45287 } }
I20260929 20:19:29.644861 9335 leader_election.cc:288] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a70c4b2e1764708a9dd9c07c329f9a3 (127.27.185.65:32983), 3bef4fa96cd24026bc828f3b6d324eba (127.27.185.67:45287)
W20260929 20:19:29.655102 9029 tablet.cc:2369] T 3585b23f63d54468989f6b915fc8ed7f P 9a70c4b2e1764708a9dd9c07c329f9a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:19:29.664821 9324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:29.645511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:46675 (local address 127.27.185.65:32983)
0929 20:19:29.645740 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:29.645750 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:29.656819 (+ 11069us) server_negotiation.cc:407] Connection header received
0929 20:19:29.657368 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:29.657374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:29.657449 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:29.657512 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:19:29.658275 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.659448 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.662388 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.663612 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.663749 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:29.663823 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:29.664415 (+ 592us) server_negotiation.cc:299] Negotiation successful
0929 20:19:29.664555 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:19:29.665726 9345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:29.645145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:32983 (local address 127.27.185.66:46675)
0929 20:19:29.656680 (+ 11535us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:29.656703 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:29.657182 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:29.657719 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:29.657723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:29.657748 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:19:29.658070 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:29.658081 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.659624 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.659629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:29.662136 (+ 2507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:29.662205 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.664027 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.664033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:29.664214 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:29.664247 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:19:29.665218 (+ 971us) client_negotiation.cc:240] Negotiation successful
0929 20:19:29.665379 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11383,"threads_started":1}
I20260929 20:19:29.672891 9326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:29.646038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:46971 (local address 127.27.185.67:45287)
0929 20:19:29.646272 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:29.646281 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:29.666230 (+ 19949us) server_negotiation.cc:407] Connection header received
0929 20:19:29.666500 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:29.666506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:29.666578 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:29.666636 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:19:29.667173 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.668241 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.670463 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.671695 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.671832 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:29.671907 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:29.672500 (+ 593us) server_negotiation.cc:299] Negotiation successful
0929 20:19:29.672639 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:19:29.673781 9345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:29.645601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:45287 (local address 127.27.185.66:46971)
0929 20:19:29.666132 (+ 20531us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:29.666152 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:29.666344 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:29.666819 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:29.666823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:29.666849 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:19:29.667000 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:29.667010 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.668454 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.668459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:29.670253 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:29.670282 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.672090 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.672096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:29.672304 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:29.672348 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 20:19:29.673310 (+ 962us) client_negotiation.cc:240] Negotiation successful
0929 20:19:29.673471 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20410,"thread_start_us":11225,"threads_started":1}
I20260929 20:19:29.674387 8982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3585b23f63d54468989f6b915fc8ed7f" candidate_uuid: "a588c33cea4a4a939a3c6ee7b0f85bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a70c4b2e1764708a9dd9c07c329f9a3" is_pre_election: true
I20260929 20:19:29.674917 9261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3585b23f63d54468989f6b915fc8ed7f" candidate_uuid: "a588c33cea4a4a939a3c6ee7b0f85bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bef4fa96cd24026bc828f3b6d324eba" is_pre_election: true
I20260929 20:19:29.675022 8982 raft_consensus.cc:2454] T 3585b23f63d54468989f6b915fc8ed7f P 9a70c4b2e1764708a9dd9c07c329f9a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a588c33cea4a4a939a3c6ee7b0f85bb3 in term 0.
I20260929 20:19:29.675407 9261 raft_consensus.cc:2454] T 3585b23f63d54468989f6b915fc8ed7f P 3bef4fa96cd24026bc828f3b6d324eba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a588c33cea4a4a939a3c6ee7b0f85bb3 in term 0.
I20260929 20:19:29.675912 9059 leader_election.cc:302] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a70c4b2e1764708a9dd9c07c329f9a3, a588c33cea4a4a939a3c6ee7b0f85bb3; no voters:
I20260929 20:19:29.676347 9335 raft_consensus.cc:2781] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:19:29.676501 9335 raft_consensus.cc:482] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:19:29.676602 9335 raft_consensus.cc:3037] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:19:29.678777 9335 raft_consensus.cc:504] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a70c4b2e1764708a9dd9c07c329f9a3" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 32983 } } peers { permanent_uuid: "a588c33cea4a4a939a3c6ee7b0f85bb3" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 35725 } } peers { permanent_uuid: "3bef4fa96cd24026bc828f3b6d324eba" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45287 } }
I20260929 20:19:29.679420 9335 leader_election.cc:288] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3 [CANDIDATE]: Term 1 election: Requested vote from peers 9a70c4b2e1764708a9dd9c07c329f9a3 (127.27.185.65:32983), 3bef4fa96cd24026bc828f3b6d324eba (127.27.185.67:45287)
I20260929 20:19:29.679839 8982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3585b23f63d54468989f6b915fc8ed7f" candidate_uuid: "a588c33cea4a4a939a3c6ee7b0f85bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a70c4b2e1764708a9dd9c07c329f9a3"
I20260929 20:19:29.680078 8982 raft_consensus.cc:3037] T 3585b23f63d54468989f6b915fc8ed7f P 9a70c4b2e1764708a9dd9c07c329f9a3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:19:29.680300 9261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3585b23f63d54468989f6b915fc8ed7f" candidate_uuid: "a588c33cea4a4a939a3c6ee7b0f85bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bef4fa96cd24026bc828f3b6d324eba"
I20260929 20:19:29.680547 9261 raft_consensus.cc:3037] T 3585b23f63d54468989f6b915fc8ed7f P 3bef4fa96cd24026bc828f3b6d324eba [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:19:29.683652 8982 raft_consensus.cc:2454] T 3585b23f63d54468989f6b915fc8ed7f P 9a70c4b2e1764708a9dd9c07c329f9a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a588c33cea4a4a939a3c6ee7b0f85bb3 in term 1.
I20260929 20:19:29.683652 9261 raft_consensus.cc:2454] T 3585b23f63d54468989f6b915fc8ed7f P 3bef4fa96cd24026bc828f3b6d324eba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a588c33cea4a4a939a3c6ee7b0f85bb3 in term 1.
I20260929 20:19:29.684190 9059 leader_election.cc:302] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3bef4fa96cd24026bc828f3b6d324eba, 9a70c4b2e1764708a9dd9c07c329f9a3, a588c33cea4a4a939a3c6ee7b0f85bb3; no voters:
I20260929 20:19:29.684536 9335 raft_consensus.cc:2781] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:19:29.685098 9335 raft_consensus.cc:686] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3 [term 1 LEADER]: Becoming Leader. State: Replica: a588c33cea4a4a939a3c6ee7b0f85bb3, State: Running, Role: LEADER
I20260929 20:19:29.685700 9335 consensus_queue.cc:237] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a70c4b2e1764708a9dd9c07c329f9a3" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 32983 } } peers { permanent_uuid: "a588c33cea4a4a939a3c6ee7b0f85bb3" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 35725 } } peers { permanent_uuid: "3bef4fa96cd24026bc828f3b6d324eba" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45287 } }
I20260929 20:19:29.690693 8824 catalog_manager.cc:5495] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a588c33cea4a4a939a3c6ee7b0f85bb3 (127.27.185.66). New cstate: current_term: 1 leader_uuid: "a588c33cea4a4a939a3c6ee7b0f85bb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a70c4b2e1764708a9dd9c07c329f9a3" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a588c33cea4a4a939a3c6ee7b0f85bb3" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 35725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bef4fa96cd24026bc828f3b6d324eba" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45287 } health_report { overall_health: UNKNOWN } } }
I20260929 20:19:29.771347 9325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:29.758334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34518 (local address 127.27.185.66:35725)
0929 20:19:29.758536 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:29.758543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:29.764080 (+ 5537us) server_negotiation.cc:407] Connection header received
0929 20:19:29.764345 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:29.764351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:29.764485 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:29.764661 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:19:29.765175 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.766222 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.768321 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.768968 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.769105 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:29.769767 (+ 662us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:19:29.770627 (+ 860us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:19:29.770785 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:29.770949 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 20:19:29.771047 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:19:29.782465 8982 raft_consensus.cc:1261] T 3585b23f63d54468989f6b915fc8ed7f P 9a70c4b2e1764708a9dd9c07c329f9a3 [term 1 FOLLOWER]: Refusing update from remote peer a588c33cea4a4a939a3c6ee7b0f85bb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:19:29.783133 9261 raft_consensus.cc:1261] T 3585b23f63d54468989f6b915fc8ed7f P 3bef4fa96cd24026bc828f3b6d324eba [term 1 FOLLOWER]: Refusing update from remote peer a588c33cea4a4a939a3c6ee7b0f85bb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:19:29.783610 9335 consensus_queue.cc:1035] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a70c4b2e1764708a9dd9c07c329f9a3" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:19:29.784245 9346 consensus_queue.cc:1035] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bef4fa96cd24026bc828f3b6d324eba" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:19:29.812748 9352 mvcc.cc:205] Tried to move back new op lower bound from 7334761143416209408 to 7334761143036989440. Current Snapshot: MvccSnapshot[applied={T|T < 7334761143416209408}]
I20260929 20:19:29.820739 9351 mvcc.cc:205] Tried to move back new op lower bound from 7334761143416209408 to 7334761143036989440. Current Snapshot: MvccSnapshot[applied={T|T < 7334761143416209408}]
I20260929 20:19:29.837107 8824 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:19:29.837407 8824 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:19:29.841797 8824 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d123993bcaf4172a8045f5e988582b6: Sending DeleteTablet for 3 replicas of tablet 3585b23f63d54468989f6b915fc8ed7f
I20260929 20:19:29.843091 8962 tablet_service.cc:1514] Processing DeleteTablet for tablet 3585b23f63d54468989f6b915fc8ed7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:19:29 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:60978
I20260929 20:19:29.843914 9241 tablet_service.cc:1514] Processing DeleteTablet for tablet 3585b23f63d54468989f6b915fc8ed7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:19:29 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:59120
I20260929 20:19:29.844156 9105 tablet_service.cc:1514] Processing DeleteTablet for tablet 3585b23f63d54468989f6b915fc8ed7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:19:29 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:34508
I20260929 20:19:29.846552 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 8862
I20260929 20:19:29.860620 9359 tablet_replica.cc:330] T 3585b23f63d54468989f6b915fc8ed7f P 9a70c4b2e1764708a9dd9c07c329f9a3: stopping tablet replica
I20260929 20:19:29.860944 9360 tablet_replica.cc:330] T 3585b23f63d54468989f6b915fc8ed7f P 3bef4fa96cd24026bc828f3b6d324eba: stopping tablet replica
I20260929 20:19:29.861169 9359 raft_consensus.cc:2229] T 3585b23f63d54468989f6b915fc8ed7f P 9a70c4b2e1764708a9dd9c07c329f9a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:19:29.861625 9360 raft_consensus.cc:2229] T 3585b23f63d54468989f6b915fc8ed7f P 3bef4fa96cd24026bc828f3b6d324eba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:19:29.861799 9359 raft_consensus.cc:2258] T 3585b23f63d54468989f6b915fc8ed7f P 9a70c4b2e1764708a9dd9c07c329f9a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:19:29.862906 9360 raft_consensus.cc:2258] T 3585b23f63d54468989f6b915fc8ed7f P 3bef4fa96cd24026bc828f3b6d324eba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:19:29.864154 9359 ts_tablet_manager.cc:1905] T 3585b23f63d54468989f6b915fc8ed7f P 9a70c4b2e1764708a9dd9c07c329f9a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:19:29.867837 9361 tablet_replica.cc:330] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3: stopping tablet replica
I20260929 20:19:29.868287 9361 raft_consensus.cc:2229] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:19:29.869017 9361 raft_consensus.cc:2258] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:19:29.871124 9361 ts_tablet_manager.cc:1905] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:19:29.871646 9359 ts_tablet_manager.cc:1918] T 3585b23f63d54468989f6b915fc8ed7f P 9a70c4b2e1764708a9dd9c07c329f9a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:19:29.871825 9359 log.cc:1196] T 3585b23f63d54468989f6b915fc8ed7f P 9a70c4b2e1764708a9dd9c07c329f9a3: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-0/wal/wals/3585b23f63d54468989f6b915fc8ed7f
I20260929 20:19:29.871973 9360 ts_tablet_manager.cc:1905] T 3585b23f63d54468989f6b915fc8ed7f P 3bef4fa96cd24026bc828f3b6d324eba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:19:29.872419 9359 ts_tablet_manager.cc:1939] T 3585b23f63d54468989f6b915fc8ed7f P 9a70c4b2e1764708a9dd9c07c329f9a3: Deleting consensus metadata
I20260929 20:19:29.874363 8812 catalog_manager.cc:4841] TS 9a70c4b2e1764708a9dd9c07c329f9a3 (127.27.185.65:32983): tablet 3585b23f63d54468989f6b915fc8ed7f (table test-table [id=2c1c10a184f04e12a0cec2a9c8a24acf]) successfully deleted
I20260929 20:19:29.878387 9361 ts_tablet_manager.cc:1918] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:19:29.878587 9361 log.cc:1196] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-1/wal/wals/3585b23f63d54468989f6b915fc8ed7f
I20260929 20:19:29.879105 9361 ts_tablet_manager.cc:1939] T 3585b23f63d54468989f6b915fc8ed7f P a588c33cea4a4a939a3c6ee7b0f85bb3: Deleting consensus metadata
I20260929 20:19:29.880987 8811 catalog_manager.cc:4841] TS a588c33cea4a4a939a3c6ee7b0f85bb3 (127.27.185.66:35725): tablet 3585b23f63d54468989f6b915fc8ed7f (table test-table [id=2c1c10a184f04e12a0cec2a9c8a24acf]) successfully deleted
I20260929 20:19:29.884052 9324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:29.848030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60992 (local address 127.27.185.65:32983)
0929 20:19:29.848215 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:29.848221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:29.859913 (+ 11692us) server_negotiation.cc:407] Connection header received
0929 20:19:29.860133 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:29.860145 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:29.860260 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:29.860384 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:29.862937 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.864066 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.866022 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:29.866664 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:29.866792 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:29.875149 (+ 8357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:29.875164 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:29.875182 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:29.875233 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:29.879448 (+ 4215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:29.880473 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:29.880484 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:29.880487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:29.880654 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:29.881164 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:29.881169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:29.881172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:29.881699 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:29.882031 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:29.883364 (+ 1333us) server_negotiation.cc:299] Negotiation successful
0929 20:19:29.883647 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:19:29.884698 9023 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:19:29.885842 9360 ts_tablet_manager.cc:1918] T 3585b23f63d54468989f6b915fc8ed7f P 3bef4fa96cd24026bc828f3b6d324eba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:19:29.886029 9360 log.cc:1196] T 3585b23f63d54468989f6b915fc8ed7f P 3bef4fa96cd24026bc828f3b6d324eba: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790712966917858-28389-0/minicluster-data/ts-2/wal/wals/3585b23f63d54468989f6b915fc8ed7f
I20260929 20:19:29.886554 9360 ts_tablet_manager.cc:1939] T 3585b23f63d54468989f6b915fc8ed7f P 3bef4fa96cd24026bc828f3b6d324eba: Deleting consensus metadata
I20260929 20:19:29.888449 8811 catalog_manager.cc:4841] TS 3bef4fa96cd24026bc828f3b6d324eba (127.27.185.67:45287): tablet 3585b23f63d54468989f6b915fc8ed7f (table test-table [id=2c1c10a184f04e12a0cec2a9c8a24acf]) successfully deleted
I20260929 20:19:30.037087 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 8862
I20260929 20:19:30.058207 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 9032
I20260929 20:19:30.072994 9325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:30.059088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34534 (local address 127.27.185.66:35725)
0929 20:19:30.059266 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:30.059272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:30.059347 (+ 75us) server_negotiation.cc:407] Connection header received
0929 20:19:30.059592 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:30.059600 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:30.059665 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:30.059789 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:30.061227 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:30.062149 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:30.063487 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:30.063910 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:30.064054 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:30.066635 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:30.066648 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:30.066663 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:30.066704 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:30.069784 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:30.070554 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:30.070562 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:30.070565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:30.070717 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:30.071131 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:30.071138 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:30.071141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:30.071624 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:30.071972 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:30.072282 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 20:19:30.072533 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:19:30.073714 9165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:19:30.186656 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 9032
I20260929 20:19:30.205518 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 9176
I20260929 20:19:30.219195 9365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:30.206275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59134 (local address 127.27.185.67:45287)
0929 20:19:30.206754 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:30.206759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:30.206775 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:19:30.206830 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:30.206833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:30.206897 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:30.207001 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:30.208426 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:30.209212 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:30.210463 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:30.210857 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:30.210950 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:30.213636 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:30.213651 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:30.213667 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:30.213702 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:30.216575 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:30.217259 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:30.217266 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:30.217268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:30.217404 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:30.217786 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:30.217789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:30.217791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:30.218131 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:30.218355 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:30.218635 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 20:19:30.218810 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":262,"threads_started":1}
I20260929 20:19:30.220000 9301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:19:30.311013 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 9176
I20260929 20:19:30.327620 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 8792
I20260929 20:19:30.328889 8855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:19:30.441797 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 8792
I20260929 20:19:30.464586 28389 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 20:19:30.803679 28389 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 20:19:30.803936 28389 mini_ranger.cc:66] Stopping Ranger...
I20260929 20:19:31.149672 28389 mini_ranger.cc:68] Stopped Ranger
2026-09-29 20:19:31.149 UTC [7838] LOG: received smart shutdown request
2026-09-29 20:19:31.153 UTC [7838] LOG: background worker "logical replication launcher" (PID 7847) exited with exit code 1
2026-09-29 20:19:31.153 UTC [7842] LOG: shutting down
2026-09-29 20:19:31.348 UTC [7838] LOG: database system is shut down
2026-09-29T20:19:31Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54084 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:19:31 dist-test-slave-4cx3 krb5kdc[9376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:19:31 dist-test-slave-4cx3 krb5kdc[9376](info): set up 2 sockets
Sep 29 20:19:31 dist-test-slave-4cx3 krb5kdc[9376](info): commencing operation
krb5kdc: starting...
W20260929 20:19:32.591073 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:19:32 dist-test-slave-4cx3 krb5kdc[9376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:19:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:19:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
I20260929 20:19:32.843770 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:38739
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:42439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:38739,127.27.185.125:37069,127.27.185.124:40453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:19:33.208619 9389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:19:33.208989 9389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:19:33.209152 9389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:19:33.216816 9389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:19:33.216979 9389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:19:33.217097 9389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:19:33.217200 9389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:19:33.217288 9389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:19:33.227560 9389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:42439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:38739,127.27.185.125:37069,127.27.185.124:40453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:38739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:19:33.228999 9389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:19:33.230387 9389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:19:33.242568 9394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:33.242758 9395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:33.243319 9397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:19:33.243741 9389 server_base.cc:1034] running on GCE node
I20260929 20:19:33.244622 9389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:19:33.246095 9389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:19:33.247354 9389 hybrid_clock.cc:648] HybridClock initialized: now 1790713173247299 us; error 41 us; skew 500 ppm
I20260929 20:19:33.247812 9389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:19:33 dist-test-slave-4cx3 krb5kdc[9376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713173, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:19:33.250996 9389 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:19:33.252701 9389 webserver.cc:460] Webserver started at http://127.27.185.126:40377/ using document root <none> and password file <none>
I20260929 20:19:33.253459 9389 fs_manager.cc:359] Metadata directory not provided
I20260929 20:19:33.253618 9389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:19:33.253937 9389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:19:33.257601 9389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "d57342817bee488a8f96e0d0d6b9b14f"
format_stamp: "Formatted at 2026-09-29 20:19:33 on dist-test-slave-4cx3"
server_key: "1fa2c65ce2d640753c226f36309b106c"
server_key_iv: "a23d2cc88961f2993a381cd5761f52a5"
server_key_version: "encryptionkey@0"
I20260929 20:19:33.258603 9389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "d57342817bee488a8f96e0d0d6b9b14f"
format_stamp: "Formatted at 2026-09-29 20:19:33 on dist-test-slave-4cx3"
server_key: "1fa2c65ce2d640753c226f36309b106c"
server_key_iv: "a23d2cc88961f2993a381cd5761f52a5"
server_key_version: "encryptionkey@0"
I20260929 20:19:33.264942 9389 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 20:19:33.268487 9404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:33.269968 9389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:19:33.270169 9389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "d57342817bee488a8f96e0d0d6b9b14f"
format_stamp: "Formatted at 2026-09-29 20:19:33 on dist-test-slave-4cx3"
server_key: "1fa2c65ce2d640753c226f36309b106c"
server_key_iv: "a23d2cc88961f2993a381cd5761f52a5"
server_key_version: "encryptionkey@0"
I20260929 20:19:33.270390 9389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:19:33.286773 9389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:19:33.291092 9389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:19:33.291460 9389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:19:33.311165 9389 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:38739
I20260929 20:19:33.311200 9455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:38739 every 8 connection(s)
I20260929 20:19:33.312649 9389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:19:33.316463 9456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:19:33.318401 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 9389
I20260929 20:19:33.318655 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:19:33.319561 28389 external_mini_cluster.cc:1428] Setting key 3588ec76c8fc6a5f1608451c1ab13a46
I20260929 20:19:33.326411 9456 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.126" port: 38739 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.185.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.125@KRBTEST.COM" created.
Sep 29 20:19:33 dist-test-slave-4cx3 krb5kdc[9376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713173, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:19:33.357407 9460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:33.330359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34214 (local address 127.27.185.126:38739)
0929 20:19:33.332423 (+ 2064us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:33.332429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:33.332480 (+ 51us) server_negotiation.cc:407] Connection header received
0929 20:19:33.334845 (+ 2365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:33.334851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:33.335085 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:33.335244 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:33.336994 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:33.338192 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:33.340551 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:33.341105 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:33.341254 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:33.348969 (+ 7715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:33.348980 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:33.348995 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:33.349033 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:33.352366 (+ 3333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:33.353439 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:33.353449 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:33.353453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:33.353611 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:33.354093 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:33.354096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:33.354098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:33.354660 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:33.354835 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:33.355932 (+ 1097us) server_negotiation.cc:299] Negotiation successful
0929 20:19:33.356303 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":118,"server-negotiator.queue_time_us":1860,"spinlock_wait_cycles":16256,"thread_start_us":1614,"threads_started":1}
I20260929 20:19:33.357467 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:33.330359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:38739 (local address 127.0.0.1:34214)
0929 20:19:33.331365 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:33.331448 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:33.334446 (+ 2998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:33.335345 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:33.335356 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:33.336236 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:19:33.336767 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:33.336795 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:33.338357 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:33.338366 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:33.340398 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:33.340417 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:33.341372 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:33.341381 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:33.341583 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:33.342305 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:19:33.342340 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:19:33.348610 (+ 6270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:19:33.352557 (+ 3947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:33.352575 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:33.352593 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:33.353263 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:33.353739 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:33.353746 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:33.353752 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:33.353979 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:33.354831 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:19:33.354844 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:19:33.355138 (+ 294us) client_negotiation.cc:769] Sending connection context
0929 20:19:33.355516 (+ 378us) client_negotiation.cc:240] Negotiation successful
0929 20:19:33.356158 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":423,"threads_started":1}
I20260929 20:19:33.360635 9456 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.125" port: 37069 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:19:33.361965 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:33.361506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:37069 (local address 127.0.0.1:39536)
0929 20:19:33.361682 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:33.361805 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.125:37069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260929 20:19:33.362313 9408 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.185.125:37069: connect: Connection refused (error 111)
I20260929 20:19:33.363933 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:33.363593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:37069 (local address 127.0.0.1:39550)
0929 20:19:33.363726 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:33.363793 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.125:37069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 20:19:33.364233 9456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.125:37069: Network error: Client connection negotiation failed: client connection to 127.27.185.125:37069: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.125.keytab.
Entry for principal kudu/127.27.185.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.125.keytab.
I20260929 20:19:33.379211 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.125:37069
--webserver_interface=127.27.185.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.125.keytab
--principal=kudu/127.27.185.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:42439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:38739,127.27.185.125:37069,127.27.185.124:40453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:19:33.413600 9456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.125" port: 37069 } attempt: 1
I20260929 20:19:33.414947 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:33.414412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:37069 (local address 127.0.0.1:39552)
0929 20:19:33.414645 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:33.414778 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.125:37069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260929 20:19:33.416008 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:33.415635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:37069 (local address 127.0.0.1:39568)
0929 20:19:33.415789 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:33.415876 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.125:37069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 20:19:33.416203 9456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.125:37069: Network error: Client connection negotiation failed: client connection to 127.27.185.125:37069: connect: Connection refused (error 111)
I20260929 20:19:33.484463 9456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.125" port: 37069 } attempt: 2
I20260929 20:19:33.485644 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:33.485153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:37069 (local address 127.0.0.1:39580)
0929 20:19:33.485415 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:33.485504 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.125:37069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 20:19:33.486735 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:33.486347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:37069 (local address 127.0.0.1:39588)
0929 20:19:33.486500 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:33.486598 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.125:37069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 20:19:33.486924 9456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.125:37069: Network error: Client connection negotiation failed: client connection to 127.27.185.125:37069: connect: Connection refused (error 111)
I20260929 20:19:33.578190 9456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.125" port: 37069 } attempt: 3
I20260929 20:19:33.579349 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:33.578882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:37069 (local address 127.0.0.1:39598)
0929 20:19:33.579121 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:33.579212 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.125:37069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260929 20:19:33.580440 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:33.580068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:37069 (local address 127.0.0.1:39608)
0929 20:19:33.580216 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:33.580307 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.125:37069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 20:19:33.580684 9456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.125:37069: Network error: Client connection negotiation failed: client connection to 127.27.185.125:37069: connect: Connection refused (error 111)
W20260929 20:19:33.716338 9463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:19:33.716707 9463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:19:33.716883 9463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 20:19:33.723937 9456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.125" port: 37069 } attempt: 4
W20260929 20:19:33.724350 9463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:19:33.724491 9463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:19:33.724587 9463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:19:33.724697 9463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:19:33.724785 9463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:19:33.725414 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:33.724764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:37069 (local address 127.0.0.1:39616)
0929 20:19:33.725131 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:33.725241 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.125:37069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
I20260929 20:19:33.726476 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:33.726072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:37069 (local address 127.0.0.1:39624)
0929 20:19:33.726222 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:33.726326 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.125:37069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 20:19:33.726666 9456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.125:37069: Network error: Client connection negotiation failed: client connection to 127.27.185.125:37069: connect: Connection refused (error 111)
I20260929 20:19:33.735754 9463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:42439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:38739,127.27.185.125:37069,127.27.185.124:40453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.125.keytab
--principal=kudu/127.27.185.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.125:37069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.185.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:19:33.737157 9463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:19:33.738539 9463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:19:33.750983 9468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:33.751001 9471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:33.750916 9469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:19:33.752331 9463 server_base.cc:1034] running on GCE node
I20260929 20:19:33.753156 9463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:19:33.754740 9463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:19:33.755956 9463 hybrid_clock.cc:648] HybridClock initialized: now 1790713173755916 us; error 34 us; skew 500 ppm
I20260929 20:19:33.756336 9463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:19:33 dist-test-slave-4cx3 krb5kdc[9376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713173, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:19:33.759701 9463 init.cc:377] Logged in from keytab as kudu/127.27.185.125@KRBTEST.COM (short username kudu)
I20260929 20:19:33.761380 9463 webserver.cc:460] Webserver started at http://127.27.185.125:35733/ using document root <none> and password file <none>
I20260929 20:19:33.762135 9463 fs_manager.cc:359] Metadata directory not provided
I20260929 20:19:33.762285 9463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:19:33.762614 9463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:19:33.765759 9463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-1/data/instance:
uuid: "bce96c7a5293471f945fcf59f7343db2"
format_stamp: "Formatted at 2026-09-29 20:19:33 on dist-test-slave-4cx3"
server_key: "8c4b0abb997e019a8308458453ca8e5e"
server_key_iv: "e14483622554b54ec303eac274032166"
server_key_version: "encryptionkey@0"
I20260929 20:19:33.766610 9463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-1/wal/instance:
uuid: "bce96c7a5293471f945fcf59f7343db2"
format_stamp: "Formatted at 2026-09-29 20:19:33 on dist-test-slave-4cx3"
server_key: "8c4b0abb997e019a8308458453ca8e5e"
server_key_iv: "e14483622554b54ec303eac274032166"
server_key_version: "encryptionkey@0"
I20260929 20:19:33.772358 9463 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:19:33.776185 9478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:33.777623 9463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:19:33.777810 9463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-1/wal
uuid: "bce96c7a5293471f945fcf59f7343db2"
format_stamp: "Formatted at 2026-09-29 20:19:33 on dist-test-slave-4cx3"
server_key: "8c4b0abb997e019a8308458453ca8e5e"
server_key_iv: "e14483622554b54ec303eac274032166"
server_key_version: "encryptionkey@0"
I20260929 20:19:33.778028 9463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:19:33.807526 9463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:19:33.811895 9463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:19:33.812184 9463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:19:33.831135 9463 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.125:37069
I20260929 20:19:33.831164 9529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.125:37069 every 8 connection(s)
I20260929 20:19:33.832602 9463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-1/data/info.pb
I20260929 20:19:33.833010 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 9463
I20260929 20:19:33.833187 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-1/wal/instance
I20260929 20:19:33.833878 28389 external_mini_cluster.cc:1428] Setting key a6612091b3542bb0a9226fae79e0a474
I20260929 20:19:33.836678 9530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:19:33.843468 9530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.126" port: 38739 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.185.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.124@KRBTEST.COM" created.
Sep 29 20:19:33 dist-test-slave-4cx3 krb5kdc[9376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713173, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.125@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:19:33.865511 9460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:33.846246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34218 (local address 127.27.185.126:38739)
0929 20:19:33.846479 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:33.846485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:33.847835 (+ 1350us) server_negotiation.cc:407] Connection header received
0929 20:19:33.849815 (+ 1980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:33.849822 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:33.849930 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:33.850082 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:33.851884 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:33.853075 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:33.854776 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:33.855232 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:33.855451 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:33.859283 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:33.859293 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:33.859300 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:33.859349 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:33.862270 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:33.863083 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:33.863093 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:33.863097 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:33.863212 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:33.863706 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:33.863711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:33.863714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:33.864010 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:33.864169 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:33.864930 (+ 761us) server_negotiation.cc:299] Negotiation successful
0929 20:19:33.865160 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 20:19:33.866578 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:33.846732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:38739 (local address 127.0.0.1:34218)
0929 20:19:33.847632 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:33.847693 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:33.849512 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:33.850341 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:33.850350 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:33.850940 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:19:33.851678 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:33.851699 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:33.853275 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:33.853279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:33.854632 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:33.854645 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:33.855602 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:33.855607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:33.855806 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:33.856544 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:19:33.856580 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:19:33.859016 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:19:33.862458 (+ 3442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:33.862469 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:33.862489 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:33.862947 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:33.863322 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:33.863328 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:33.863332 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:33.863584 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:33.864139 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:19:33.864147 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:19:33.864546 (+ 399us) client_negotiation.cc:769] Sending connection context
0929 20:19:33.864819 (+ 273us) client_negotiation.cc:240] Negotiation successful
0929 20:19:33.865367 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":392,"threads_started":1}
I20260929 20:19:33.868364 9530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.125" port: 37069 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:19:33 dist-test-slave-4cx3 krb5kdc[9376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713173, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.125@KRBTEST.COM for kudu/127.27.185.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.124.keytab.
Entry for principal kudu/127.27.185.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.124.keytab.
I20260929 20:19:33.887024 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:33.869226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:37069 (local address 127.0.0.1:39640)
0929 20:19:33.869471 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:33.869493 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:33.869607 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:33.870543 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:33.870555 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:33.871087 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:19:33.871292 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:33.871306 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:33.872733 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:33.872741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:33.874394 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:33.874413 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:33.875518 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:33.875531 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:33.875763 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:33.876392 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:19:33.876410 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:19:33.880697 (+ 4287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:19:33.884388 (+ 3691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:33.884394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:33.884398 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:33.884808 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:33.885172 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:33.885179 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:33.885184 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:33.885372 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:33.885982 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:19:33.885992 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:19:33.886132 (+ 140us) client_negotiation.cc:769] Sending connection context
0929 20:19:33.886286 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 20:19:33.886591 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":23}
I20260929 20:19:33.887079 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:33.869393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39640 (local address 127.27.185.125:37069)
0929 20:19:33.870119 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:33.870125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:33.870149 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:19:33.870196 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:33.870202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:33.870376 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:33.870497 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:33.871507 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:33.872582 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:33.874550 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:33.875345 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:33.875509 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:33.880931 (+ 5422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:33.880945 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:33.880961 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:33.880989 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:33.884233 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:33.884931 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:33.884946 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:33.884949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:33.885063 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:33.885482 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:33.885489 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:33.885493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:33.885860 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:33.886084 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:33.886433 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 20:19:33.886700 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":575,"thread_start_us":434,"threads_started":1}
I20260929 20:19:33.888211 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.124:40453
--webserver_interface=127.27.185.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.124.keytab
--principal=kudu/127.27.185.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:42439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:38739,127.27.185.125:37069,127.27.185.124:40453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:19:33.889605 9530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.124" port: 40453 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:19:33.890877 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:33.890277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:40453 (local address 127.0.0.1:39332)
0929 20:19:33.890523 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:33.890658 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:40453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 20:19:33.891157 9481 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.185.124:40453: connect: Connection refused (error 111)
I20260929 20:19:33.892994 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:33.892506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:40453 (local address 127.0.0.1:39340)
0929 20:19:33.892729 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:33.892829 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:40453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260929 20:19:33.893206 9530 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.124:40453: Network error: Client connection negotiation failed: client connection to 127.27.185.124:40453: connect: Connection refused (error 111)
I20260929 20:19:33.942564 9530 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.124" port: 40453 } attempt: 1
I20260929 20:19:33.943893 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:33.943450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:40453 (local address 127.0.0.1:39344)
0929 20:19:33.943640 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:33.943745 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:40453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 20:19:33.945039 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:33.944655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:40453 (local address 127.0.0.1:39356)
0929 20:19:33.944802 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:33.944890 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:40453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 20:19:33.945253 9530 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.124:40453: Network error: Client connection negotiation failed: client connection to 127.27.185.124:40453: connect: Connection refused (error 111)
I20260929 20:19:34.013504 9530 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.124" port: 40453 } attempt: 2
I20260929 20:19:34.014789 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.014248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:40453 (local address 127.0.0.1:39366)
0929 20:19:34.014480 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:34.014599 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:40453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 20:19:34.016057 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.015619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:40453 (local address 127.0.0.1:39376)
0929 20:19:34.015774 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:34.015862 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:40453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 20:19:34.016286 9530 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.124:40453: Network error: Client connection negotiation failed: client connection to 127.27.185.124:40453: connect: Connection refused (error 111)
I20260929 20:19:34.025931 9456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.125" port: 37069 } attempt: 5
Sep 29 20:19:34 dist-test-slave-4cx3 krb5kdc[9376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713173, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for kudu/127.27.185.125@KRBTEST.COM
I20260929 20:19:34.040747 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.026711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:37069 (local address 127.0.0.1:39650)
0929 20:19:34.026952 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:34.026970 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:34.027086 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:34.027543 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:34.027547 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:34.027924 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:19:34.028124 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:34.028133 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.029678 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.029683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:34.030557 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:34.030569 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.031665 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.031670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:34.031812 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:34.032462 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:19:34.032485 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:19:34.034480 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:19:34.038290 (+ 3810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:34.038294 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:34.038297 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:34.038678 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:34.039134 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:34.039138 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:34.039140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:34.039288 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:34.039983 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:19:34.039987 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:19:34.040128 (+ 141us) client_negotiation.cc:769] Sending connection context
0929 20:19:34.040248 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 20:19:34.040497 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 20:19:34.040966 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.026829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39650 (local address 127.27.185.125:37069)
0929 20:19:34.027067 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:34.027076 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:34.027105 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:19:34.027204 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:34.027212 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:34.027308 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:34.027481 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:34.028303 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.029533 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.030705 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.031518 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.031656 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:34.034689 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:34.034695 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:34.034699 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:34.034738 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:34.038134 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:34.038801 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:34.038807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:34.038811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:34.038958 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:34.039429 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:34.039434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:34.039437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:34.039845 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:34.039990 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:34.040409 (+ 419us) server_negotiation.cc:299] Negotiation successful
0929 20:19:34.040654 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 20:19:34.042147 9456 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.124" port: 40453 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:19:34.043254 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.042870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:40453 (local address 127.0.0.1:39384)
0929 20:19:34.043043 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:34.043128 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:40453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 20:19:34.044271 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.043862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:40453 (local address 127.0.0.1:39386)
0929 20:19:34.044036 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:34.044137 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:40453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 20:19:34.044497 9456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.124:40453: Network error: Client connection negotiation failed: client connection to 127.27.185.124:40453: connect: Connection refused (error 111)
I20260929 20:19:34.095754 9456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.124" port: 40453 } attempt: 1
I20260929 20:19:34.097087 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.096509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:40453 (local address 127.0.0.1:39396)
0929 20:19:34.096742 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:34.096864 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:40453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 20:19:34.098151 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.097762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:40453 (local address 127.0.0.1:39406)
0929 20:19:34.097923 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:34.098021 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:40453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 20:19:34.098382 9456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.124:40453: Network error: Client connection negotiation failed: client connection to 127.27.185.124:40453: connect: Connection refused (error 111)
I20260929 20:19:34.107491 9530 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.124" port: 40453 } attempt: 3
I20260929 20:19:34.108693 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.108164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:40453 (local address 127.0.0.1:39412)
0929 20:19:34.108384 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:34.108496 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:40453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 20:19:34.109872 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.109396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:40453 (local address 127.0.0.1:39426)
0929 20:19:34.109639 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:34.109732 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:40453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
W20260929 20:19:34.110188 9530 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.124:40453: Network error: Client connection negotiation failed: client connection to 127.27.185.124:40453: connect: Connection refused (error 111)
I20260929 20:19:34.166636 9456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.124" port: 40453 } attempt: 2
I20260929 20:19:34.167951 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.167368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:40453 (local address 127.0.0.1:39432)
0929 20:19:34.167683 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:34.167783 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:40453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260929 20:19:34.169064 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.168693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:40453 (local address 127.0.0.1:39444)
0929 20:19:34.168857 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:34.168949 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:40453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 20:19:34.169282 9456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.124:40453: Network error: Client connection negotiation failed: client connection to 127.27.185.124:40453: connect: Connection refused (error 111)
W20260929 20:19:34.228835 9537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:19:34.229197 9537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:19:34.229377 9537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:19:34.237360 9537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:19:34.237527 9537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:19:34.237632 9537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:19:34.237744 9537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:19:34.237813 9537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:19:34.248031 9537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:42439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:38739,127.27.185.125:37069,127.27.185.124:40453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.124.keytab
--principal=kudu/127.27.185.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.124:40453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.185.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:19:34.249455 9537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:19:34.250797 9537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 20:19:34.253454 9530 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.124" port: 40453 } attempt: 4
I20260929 20:19:34.254663 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.254182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:40453 (local address 127.0.0.1:39448)
0929 20:19:34.254419 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:34.254517 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:40453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 20:19:34.255821 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.255367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:40453 (local address 127.0.0.1:39450)
0929 20:19:34.255640 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:34.255712 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:40453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
W20260929 20:19:34.256071 9530 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.124:40453: Network error: Client connection negotiation failed: client connection to 127.27.185.124:40453: connect: Connection refused (error 111)
W20260929 20:19:34.263454 9545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:34.263877 9542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:34.264145 9543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:19:34.264519 9537 server_base.cc:1034] running on GCE node
I20260929 20:19:34.265538 9537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:19:34.267017 9537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:19:34.268244 9537 hybrid_clock.cc:648] HybridClock initialized: now 1790713174268195 us; error 45 us; skew 500 ppm
I20260929 20:19:34.268606 9537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:19:34 dist-test-slave-4cx3 krb5kdc[9376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713174, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:19:34.271849 9537 init.cc:377] Logged in from keytab as kudu/127.27.185.124@KRBTEST.COM (short username kudu)
I20260929 20:19:34.273381 9537 webserver.cc:460] Webserver started at http://127.27.185.124:35739/ using document root <none> and password file <none>
I20260929 20:19:34.274211 9537 fs_manager.cc:359] Metadata directory not provided
I20260929 20:19:34.274356 9537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:19:34.274680 9537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:19:34.275532 9456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.124" port: 40453 } attempt: 3
I20260929 20:19:34.276597 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.276110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:40453 (local address 127.0.0.1:39460)
0929 20:19:34.276331 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:34.276442 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:40453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 20:19:34.277819 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.277307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:40453 (local address 127.0.0.1:39476)
0929 20:19:34.277584 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:34.277682 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:40453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
W20260929 20:19:34.278086 9456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.124:40453: Network error: Client connection negotiation failed: client connection to 127.27.185.124:40453: connect: Connection refused (error 111)
I20260929 20:19:34.278383 9537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-2/data/instance:
uuid: "7228480d03d7445989d1633de8d1c86f"
format_stamp: "Formatted at 2026-09-29 20:19:34 on dist-test-slave-4cx3"
server_key: "e83863d567d9e9a6b89046d02850ca44"
server_key_iv: "8cd648f6c3e5af36ca17d76147a9be1f"
server_key_version: "encryptionkey@0"
I20260929 20:19:34.279147 9537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-2/wal/instance:
uuid: "7228480d03d7445989d1633de8d1c86f"
format_stamp: "Formatted at 2026-09-29 20:19:34 on dist-test-slave-4cx3"
server_key: "e83863d567d9e9a6b89046d02850ca44"
server_key_iv: "8cd648f6c3e5af36ca17d76147a9be1f"
server_key_version: "encryptionkey@0"
I20260929 20:19:34.284850 9537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:19:34.288335 9552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:34.289742 9537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:19:34.289880 9537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-2/wal
uuid: "7228480d03d7445989d1633de8d1c86f"
format_stamp: "Formatted at 2026-09-29 20:19:34 on dist-test-slave-4cx3"
server_key: "e83863d567d9e9a6b89046d02850ca44"
server_key_iv: "8cd648f6c3e5af36ca17d76147a9be1f"
server_key_version: "encryptionkey@0"
I20260929 20:19:34.290050 9537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:19:34.306846 9537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:19:34.311172 9537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:19:34.311522 9537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:19:34.331437 9537 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.124:40453
I20260929 20:19:34.331492 9603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.124:40453 every 8 connection(s)
I20260929 20:19:34.332818 9537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-2/data/info.pb
I20260929 20:19:34.336444 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:19:34.342290 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 9537
I20260929 20:19:34.342504 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790712966917858-28389-0/minicluster-data/master-2/wal/instance
I20260929 20:19:34.343083 28389 external_mini_cluster.cc:1428] Setting key c21249ff4df3c38c92ba6cfa027ae06e
I20260929 20:19:34.343237 9604 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.126" port: 38739 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:19:34 dist-test-slave-4cx3 krb5kdc[9376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713174, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.124@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
Sep 29 20:19:34 dist-test-slave-4cx3 krb5kdc[9376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.124@KRBTEST.COM
I20260929 20:19:34.363830 9460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.346408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34228 (local address 127.27.185.126:38739)
0929 20:19:34.346592 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:34.346597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:34.347541 (+ 944us) server_negotiation.cc:407] Connection header received
0929 20:19:34.348688 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:34.348694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:34.348780 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:34.348891 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:34.350172 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.350864 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.352425 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.352849 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.352995 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:34.356779 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:34.356785 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:34.356788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:34.356823 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:34.360221 (+ 3398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:34.360996 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:34.361005 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:34.361010 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:34.361186 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:34.361643 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:34.361648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:34.361651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:34.361953 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:34.362124 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:34.363176 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0929 20:19:34.363439 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:19:34.365067 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.346451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:38739 (local address 127.0.0.1:34228)
0929 20:19:34.347322 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:34.347359 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:34.348381 (+ 1022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:34.348961 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:34.348969 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:34.349621 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:19:34.349995 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:34.350016 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.351042 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.351046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:34.352276 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:34.352288 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.353102 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.353110 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:34.353369 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:34.354075 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:19:34.354098 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:19:34.356491 (+ 2393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:19:34.360359 (+ 3868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:34.360369 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:34.360379 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:34.360850 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:34.361303 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:34.361307 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:34.361309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:34.361517 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:34.362127 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:19:34.362138 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:19:34.362649 (+ 511us) client_negotiation.cc:769] Sending connection context
0929 20:19:34.363023 (+ 374us) client_negotiation.cc:240] Negotiation successful
0929 20:19:34.363498 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":4,"spinlock_wait_cycles":17280,"thread_start_us":495,"threads_started":1}
I20260929 20:19:34.365425 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.345049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39492 (local address 127.27.185.124:40453)
0929 20:19:34.345846 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:34.345860 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:34.345910 (+ 50us) server_negotiation.cc:407] Connection header received
0929 20:19:34.348337 (+ 2427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:34.348359 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:34.348612 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:34.348823 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:34.350392 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.351627 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.352965 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.353579 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.353713 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:34.358189 (+ 4476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:34.358201 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:34.358219 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:34.358257 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:34.361634 (+ 3377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:34.362377 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:34.362383 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:34.362385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:34.362504 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:34.362950 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:34.362953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:34.362955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:34.363422 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:34.363737 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:34.364702 (+ 965us) server_negotiation.cc:299] Negotiation successful
0929 20:19:34.364949 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":589,"thread_start_us":385,"threads_started":1}
I20260929 20:19:34.366936 9604 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.125" port: 37069 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:19:34 dist-test-slave-4cx3 krb5kdc[9376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713174, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.124@KRBTEST.COM for kudu/127.27.185.125@KRBTEST.COM
Sep 29 20:19:34 dist-test-slave-4cx3 krb5kdc[9376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:19:34.381388 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.367824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39652 (local address 127.27.185.125:37069)
0929 20:19:34.368035 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:34.368041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:34.368171 (+ 130us) server_negotiation.cc:407] Connection header received
0929 20:19:34.368293 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:34.368300 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:34.368382 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:34.368536 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:34.369271 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.370008 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.371324 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.371843 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.371938 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:34.375407 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:34.375418 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:34.375424 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:34.375471 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:34.378540 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:34.379348 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:34.379353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:34.379355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:34.379564 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:34.379991 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:34.379994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:34.379997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:34.380356 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:34.380467 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:34.380954 (+ 487us) server_negotiation.cc:299] Negotiation successful
0929 20:19:34.381119 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:19:34.381565 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.367879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:37069 (local address 127.0.0.1:39652)
0929 20:19:34.368091 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:34.368106 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:34.368199 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:34.368577 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:34.368585 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:34.368956 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:19:34.369120 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:34.369128 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.370150 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.370156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:34.371170 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:34.371182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.371993 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.371999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:34.372201 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:34.372775 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:19:34.372801 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:19:34.375164 (+ 2363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:19:34.378680 (+ 3516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:34.378686 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:34.378690 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:34.379229 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:34.379680 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:34.379686 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:34.379690 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:34.379878 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:34.380511 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:19:34.380518 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:19:34.380700 (+ 182us) client_negotiation.cc:769] Sending connection context
0929 20:19:34.380879 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 20:19:34.381197 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260929 20:19:34.382846 9604 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.124" port: 40453 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:19:34.384747 9460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.368674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34230 (local address 127.27.185.126:38739)
0929 20:19:34.368832 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:34.368839 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:34.368876 (+ 37us) server_negotiation.cc:407] Connection header received
0929 20:19:34.369067 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:34.369072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:34.369144 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:34.369250 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:34.370536 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.371291 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.372893 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.373300 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.373390 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:34.378391 (+ 5001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:34.378397 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:34.378400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:34.378432 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:34.381902 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:34.382601 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:34.382607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:34.382611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:34.382786 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:34.383256 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:34.383264 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:34.383269 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:34.383608 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:34.383903 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:34.384157 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 20:19:34.384402 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 20:19:34 dist-test-slave-4cx3 krb5kdc[9376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713174, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.124@KRBTEST.COM for kudu/127.27.185.124@KRBTEST.COM
Sep 29 20:19:34 dist-test-slave-4cx3 krb5kdc[9376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.125@KRBTEST.COM
I20260929 20:19:34.396692 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.383629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:40453 (local address 127.0.0.1:39498)
0929 20:19:34.383815 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:34.383833 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:34.383926 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:34.384271 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:34.384276 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:34.384729 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:19:34.384909 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:34.384922 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.386049 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.386055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:34.387231 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:34.387242 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.387979 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.387985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:34.388167 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:34.388725 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:19:34.388750 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:19:34.391043 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:19:34.394258 (+ 3215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:34.394265 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:34.394272 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:34.394729 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:34.395175 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:34.395180 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:34.395183 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:34.395344 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:34.395882 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:19:34.395889 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:19:34.396066 (+ 177us) client_negotiation.cc:769] Sending connection context
0929 20:19:34.396208 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 20:19:34.396412 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 20:19:34.396966 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.383848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39498 (local address 127.27.185.124:40453)
0929 20:19:34.383995 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:34.384002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:34.384020 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:19:34.384069 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:34.384073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:34.384141 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:34.384226 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:34.385064 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.385925 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.387366 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.387831 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.387963 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:34.391228 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:34.391233 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:34.391236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:34.391263 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:34.394136 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:34.394852 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:34.394858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:34.394862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:34.395017 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:34.395478 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:34.395483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:34.395486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:34.395754 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:34.395864 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:34.396337 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 20:19:34.396579 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 20:19:34.401810 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.386471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39664 (local address 127.27.185.125:37069)
0929 20:19:34.386638 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:34.386645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:34.386663 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:19:34.386764 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:34.386769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:34.386833 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:34.386929 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:34.388193 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.389297 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.390674 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.391324 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.391492 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:34.396140 (+ 4648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:34.396145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:34.396148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:34.396179 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:34.399150 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:34.399918 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:34.399930 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:34.399933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:34.400062 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:34.400430 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:34.400439 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:34.400442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:34.400708 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:34.400974 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:34.401267 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 20:19:34.401490 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:19:34.401847 9604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f: Bootstrap starting.
I20260929 20:19:34.405735 9604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f: Neither blocks nor log segments found. Creating new log.
I20260929 20:19:34.406446 28389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:19:34.406911 9604 log.cc:824] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:19:34.409739 9604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f: No bootstrap required, opened a new log
I20260929 20:19:34.420598 9604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57342817bee488a8f96e0d0d6b9b14f" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 38739 } } peers { permanent_uuid: "bce96c7a5293471f945fcf59f7343db2" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 37069 } } peers { permanent_uuid: "7228480d03d7445989d1633de8d1c86f" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 40453 } }
I20260929 20:19:34.421255 9604 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:19:34.421469 9604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7228480d03d7445989d1633de8d1c86f, State: Initialized, Role: FOLLOWER
I20260929 20:19:34.422359 9604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57342817bee488a8f96e0d0d6b9b14f" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 38739 } } peers { permanent_uuid: "bce96c7a5293471f945fcf59f7343db2" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 37069 } } peers { permanent_uuid: "7228480d03d7445989d1633de8d1c86f" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 40453 } }
I20260929 20:19:34.429584 9617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57342817bee488a8f96e0d0d6b9b14f" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 38739 } } peers { permanent_uuid: "bce96c7a5293471f945fcf59f7343db2" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 37069 } } peers { permanent_uuid: "7228480d03d7445989d1633de8d1c86f" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 40453 } } }
I20260929 20:19:34.432204 9617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:19:34.432070 9604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:19:34.432950 9460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.411812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34242 (local address 127.27.185.126:38739)
0929 20:19:34.412013 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:34.412021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:34.412649 (+ 628us) server_negotiation.cc:407] Connection header received
0929 20:19:34.412955 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:34.412971 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:34.413056 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:34.413131 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:34.415852 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.417008 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.419375 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.420033 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.420140 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:34.424352 (+ 4212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:34.424362 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:34.424369 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:34.424440 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:34.429055 (+ 4615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:34.430377 (+ 1322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:34.430382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:34.430386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:34.430607 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:34.431100 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:34.431105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:34.431110 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:34.431508 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:34.431732 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:34.432328 (+ 596us) server_negotiation.cc:299] Negotiation successful
0929 20:19:34.432590 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:19:34.439821 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.411918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39680 (local address 127.27.185.125:37069)
0929 20:19:34.412130 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:34.412139 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:34.419427 (+ 7288us) server_negotiation.cc:407] Connection header received
0929 20:19:34.419986 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:34.419996 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:34.420102 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:34.420194 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:34.423312 (+ 3118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.424518 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.426713 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.427340 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.427466 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:34.431210 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:34.431216 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:34.431223 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:34.431273 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:34.436315 (+ 5042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:34.437224 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:34.437232 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:34.437237 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:34.437399 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:34.437875 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:34.437881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:34.437885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:34.438248 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:34.438478 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:34.439095 (+ 617us) server_negotiation.cc:299] Negotiation successful
0929 20:19:34.439340 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:19:34.442755 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.413070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39504 (local address 127.27.185.124:40453)
0929 20:19:34.413403 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:34.413409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:34.424185 (+ 10776us) server_negotiation.cc:407] Connection header received
0929 20:19:34.424542 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:34.424548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:34.424636 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:34.424710 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:34.426593 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.427674 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.430078 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.430848 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.430949 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:34.434953 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:34.434962 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:34.434967 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:34.435009 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:34.438928 (+ 3919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:34.439852 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:34.439857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:34.439863 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:34.440074 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:34.440522 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:34.440527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:34.440531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:34.440923 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:34.442164 (+ 1241us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:34.442292 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 20:19:34.442465 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
W20260929 20:19:34.446403 9628 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:19:34.446549 9628 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:19:34.441269 9537 master_runner.cc:186] Error getting master registration for 127.27.185.126:38739: OK, instance_id { permanent_uuid: "d57342817bee488a8f96e0d0d6b9b14f" instance_seqno: 1790713173305522 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 20:19:34.451939 9537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 20:19:34.455453 9456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.124" port: 40453 } attempt: 4
Sep 29 20:19:34 dist-test-slave-4cx3 krb5kdc[9376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713173, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for kudu/127.27.185.124@KRBTEST.COM
I20260929 20:19:34.470934 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.456249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39510 (local address 127.27.185.124:40453)
0929 20:19:34.456426 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:34.456435 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:34.456602 (+ 167us) server_negotiation.cc:407] Connection header received
0929 20:19:34.456887 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:34.456893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:34.457011 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:34.457145 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:34.458159 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.458930 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.460745 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.461214 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.461323 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:34.465106 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:34.465112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:34.465116 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:34.465168 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:34.468125 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:34.468843 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:34.468850 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:34.468853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:34.469001 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:34.469425 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:34.469430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:34.469435 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:34.469768 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:34.469927 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:34.470318 (+ 391us) server_negotiation.cc:299] Negotiation successful
0929 20:19:34.470557 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:19:34.470952 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.456074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:40453 (local address 127.0.0.1:39510)
0929 20:19:34.456349 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:34.456484 (+ 135us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:34.456607 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:34.457187 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:34.457194 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:34.457745 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:19:34.457967 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:34.457981 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.459086 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.459092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:34.460598 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:34.460613 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.461377 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.461383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:34.461583 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:34.462289 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:19:34.462308 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:19:34.464882 (+ 2574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:19:34.468277 (+ 3395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:34.468282 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:34.468287 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:34.468713 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:34.469120 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:34.469125 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:34.469130 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:34.469297 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:34.469892 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:19:34.469897 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:19:34.470071 (+ 174us) client_negotiation.cc:769] Sending connection context
0929 20:19:34.470225 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 20:19:34.470539 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260929 20:19:34.476365 9456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d57342817bee488a8f96e0d0d6b9b14f: Bootstrap starting.
I20260929 20:19:34.479682 9456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d57342817bee488a8f96e0d0d6b9b14f: Neither blocks nor log segments found. Creating new log.
I20260929 20:19:34.480738 9456 log.cc:824] T 00000000000000000000000000000000 P d57342817bee488a8f96e0d0d6b9b14f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:19:34.484157 9456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d57342817bee488a8f96e0d0d6b9b14f: No bootstrap required, opened a new log
I20260929 20:19:34.491915 9456 raft_consensus.cc:348] T 00000000000000000000000000000000 P d57342817bee488a8f96e0d0d6b9b14f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57342817bee488a8f96e0d0d6b9b14f" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 38739 } } peers { permanent_uuid: "bce96c7a5293471f945fcf59f7343db2" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 37069 } } peers { permanent_uuid: "7228480d03d7445989d1633de8d1c86f" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 40453 } }
I20260929 20:19:34.492295 9456 raft_consensus.cc:374] T 00000000000000000000000000000000 P d57342817bee488a8f96e0d0d6b9b14f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:19:34.492448 9456 raft_consensus.cc:729] T 00000000000000000000000000000000 P d57342817bee488a8f96e0d0d6b9b14f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d57342817bee488a8f96e0d0d6b9b14f, State: Initialized, Role: FOLLOWER
I20260929 20:19:34.493121 9456 consensus_queue.cc:260] T 00000000000000000000000000000000 P d57342817bee488a8f96e0d0d6b9b14f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57342817bee488a8f96e0d0d6b9b14f" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 38739 } } peers { permanent_uuid: "bce96c7a5293471f945fcf59f7343db2" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 37069 } } peers { permanent_uuid: "7228480d03d7445989d1633de8d1c86f" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 40453 } }
I20260929 20:19:34.494190 9631 sys_catalog.cc:455] T 00000000000000000000000000000000 P d57342817bee488a8f96e0d0d6b9b14f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57342817bee488a8f96e0d0d6b9b14f" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 38739 } } peers { permanent_uuid: "bce96c7a5293471f945fcf59f7343db2" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 37069 } } peers { permanent_uuid: "7228480d03d7445989d1633de8d1c86f" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 40453 } } }
I20260929 20:19:34.494630 9631 sys_catalog.cc:458] T 00000000000000000000000000000000 P d57342817bee488a8f96e0d0d6b9b14f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:19:34.495574 9456 sys_catalog.cc:564] T 00000000000000000000000000000000 P d57342817bee488a8f96e0d0d6b9b14f [sys.catalog]: configured and running, proceeding with master startup.
W20260929 20:19:34.506472 9642 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d57342817bee488a8f96e0d0d6b9b14f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:19:34.506731 9642 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:19:34.505476 9389 master_runner.cc:186] Error getting master registration for 127.27.185.125:37069: OK, instance_id { permanent_uuid: "bce96c7a5293471f945fcf59f7343db2" instance_seqno: 1790713173825451 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 20:19:34.520612 9389 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 20:19:34.542609 9617 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:19:34.542980 9617 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57342817bee488a8f96e0d0d6b9b14f" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 38739 } } peers { permanent_uuid: "bce96c7a5293471f945fcf59f7343db2" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 37069 } } peers { permanent_uuid: "7228480d03d7445989d1633de8d1c86f" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 40453 } }
W20260929 20:19:34.544449 9410 tablet.cc:2369] T 00000000000000000000000000000000 P d57342817bee488a8f96e0d0d6b9b14f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:19:34.544693 9617 leader_election.cc:288] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d57342817bee488a8f96e0d0d6b9b14f (127.27.185.126:38739), bce96c7a5293471f945fcf59f7343db2 (127.27.185.125:37069)
I20260929 20:19:34.546013 9431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7228480d03d7445989d1633de8d1c86f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d57342817bee488a8f96e0d0d6b9b14f" is_pre_election: true
I20260929 20:19:34.546746 9431 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d57342817bee488a8f96e0d0d6b9b14f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7228480d03d7445989d1633de8d1c86f in term 0.
I20260929 20:19:34.547504 9555 leader_election.cc:302] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7228480d03d7445989d1633de8d1c86f, d57342817bee488a8f96e0d0d6b9b14f; no voters:
I20260929 20:19:34.547953 9617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:19:34.548112 9617 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:19:34.548245 9617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:19:34.551712 9617 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57342817bee488a8f96e0d0d6b9b14f" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 38739 } } peers { permanent_uuid: "bce96c7a5293471f945fcf59f7343db2" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 37069 } } peers { permanent_uuid: "7228480d03d7445989d1633de8d1c86f" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 40453 } }
I20260929 20:19:34.552436 9617 leader_election.cc:288] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f [CANDIDATE]: Term 1 election: Requested vote from peers d57342817bee488a8f96e0d0d6b9b14f (127.27.185.126:38739), bce96c7a5293471f945fcf59f7343db2 (127.27.185.125:37069)
I20260929 20:19:34.553056 9431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7228480d03d7445989d1633de8d1c86f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d57342817bee488a8f96e0d0d6b9b14f"
I20260929 20:19:34.553288 9431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d57342817bee488a8f96e0d0d6b9b14f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:19:34.555317 9530 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.124" port: 40453 } attempt: 5
I20260929 20:19:34.559243 9431 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d57342817bee488a8f96e0d0d6b9b14f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7228480d03d7445989d1633de8d1c86f in term 1.
I20260929 20:19:34.561933 9555 leader_election.cc:302] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7228480d03d7445989d1633de8d1c86f, d57342817bee488a8f96e0d0d6b9b14f; no voters:
I20260929 20:19:34.562827 9617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f [term 1 FOLLOWER]: Leader election won for term 1
W20260929 20:19:34.563900 9558 tablet.cc:2369] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 20:19:34 dist-test-slave-4cx3 krb5kdc[9376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713173, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.125@KRBTEST.COM for kudu/127.27.185.124@KRBTEST.COM
I20260929 20:19:34.578189 9617 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f [term 1 LEADER]: Becoming Leader. State: Replica: 7228480d03d7445989d1633de8d1c86f, State: Running, Role: LEADER
I20260929 20:19:34.579638 9617 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57342817bee488a8f96e0d0d6b9b14f" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 38739 } } peers { permanent_uuid: "bce96c7a5293471f945fcf59f7343db2" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 37069 } } peers { permanent_uuid: "7228480d03d7445989d1633de8d1c86f" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 40453 } }
I20260929 20:19:34.582787 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.556749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:40453 (local address 127.0.0.1:39518)
0929 20:19:34.556947 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:34.556968 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:34.557085 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:34.557679 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:34.557684 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:34.558255 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:19:34.558454 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:34.558466 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.560088 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.560097 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:34.561313 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:34.561325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.562480 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.562486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:34.562746 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:34.563493 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:19:34.563518 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:19:34.568362 (+ 4844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:19:34.575595 (+ 7233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:34.575602 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:34.575607 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:34.577969 (+ 2362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:34.579006 (+ 1037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:34.579035 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:34.579039 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:34.579414 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:34.580711 (+ 1297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:19:34.580716 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:19:34.581027 (+ 311us) client_negotiation.cc:769] Sending connection context
0929 20:19:34.581352 (+ 325us) client_negotiation.cc:240] Negotiation successful
0929 20:19:34.582010 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 20:19:34.584990 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:34.557161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39518 (local address 127.27.185.124:40453)
0929 20:19:34.557314 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:34.557322 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:34.557340 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:19:34.557394 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:34.557399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:34.557485 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:34.557579 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:34.558647 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.559822 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.561469 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:34.562126 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:34.562261 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:34.568691 (+ 6430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:34.568703 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:34.568714 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:34.568775 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:34.575240 (+ 6465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:34.578488 (+ 3248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:34.578496 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:34.578500 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:34.578781 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:34.579710 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:34.579716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:34.579720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:34.580468 (+ 748us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:34.583478 (+ 3010us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:34.583725 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 20:19:34.584253 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:19:34.593235 9530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bce96c7a5293471f945fcf59f7343db2: Bootstrap starting.
I20260929 20:19:34.603127 9530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bce96c7a5293471f945fcf59f7343db2: Neither blocks nor log segments found. Creating new log.
I20260929 20:19:34.607621 9530 log.cc:824] T 00000000000000000000000000000000 P bce96c7a5293471f945fcf59f7343db2: Log is configured to *not* fsync() on all Append() calls
I20260929 20:19:34.607928 9644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7228480d03d7445989d1633de8d1c86f. Latest consensus state: current_term: 1 leader_uuid: "7228480d03d7445989d1633de8d1c86f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57342817bee488a8f96e0d0d6b9b14f" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 38739 } } peers { permanent_uuid: "bce96c7a5293471f945fcf59f7343db2" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 37069 } } peers { permanent_uuid: "7228480d03d7445989d1633de8d1c86f" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 40453 } } }
I20260929 20:19:34.608474 9644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f [sys.catalog]: This master's current role is: LEADER
I20260929 20:19:34.614778 9530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bce96c7a5293471f945fcf59f7343db2: No bootstrap required, opened a new log
I20260929 20:19:34.616003 9647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:19:34.618332 9647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:19:34.627862 9530 raft_consensus.cc:348] T 00000000000000000000000000000000 P bce96c7a5293471f945fcf59f7343db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57342817bee488a8f96e0d0d6b9b14f" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 38739 } } peers { permanent_uuid: "bce96c7a5293471f945fcf59f7343db2" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 37069 } } peers { permanent_uuid: "7228480d03d7445989d1633de8d1c86f" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 40453 } }
I20260929 20:19:34.628351 9530 raft_consensus.cc:374] T 00000000000000000000000000000000 P bce96c7a5293471f945fcf59f7343db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:19:34.628481 9530 raft_consensus.cc:729] T 00000000000000000000000000000000 P bce96c7a5293471f945fcf59f7343db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bce96c7a5293471f945fcf59f7343db2, State: Initialized, Role: FOLLOWER
I20260929 20:19:34.629266 9530 consensus_queue.cc:260] T 00000000000000000000000000000000 P bce96c7a5293471f945fcf59f7343db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57342817bee488a8f96e0d0d6b9b14f" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 38739 } } peers { permanent_uuid: "bce96c7a5293471f945fcf59f7343db2" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 37069 } } peers { permanent_uuid: "7228480d03d7445989d1633de8d1c86f" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 40453 } }
I20260929 20:19:34.637750 9431 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d57342817bee488a8f96e0d0d6b9b14f [term 1 FOLLOWER]: Refusing update from remote peer 7228480d03d7445989d1633de8d1c86f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:19:34.639372 9644 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d57342817bee488a8f96e0d0d6b9b14f" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:19:34.641978 9530 sys_catalog.cc:564] T 00000000000000000000000000000000 P bce96c7a5293471f945fcf59f7343db2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:19:34.645447 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7228480d03d7445989d1633de8d1c86f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce96c7a5293471f945fcf59f7343db2"
I20260929 20:19:34.646368 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7228480d03d7445989d1633de8d1c86f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce96c7a5293471f945fcf59f7343db2" is_pre_election: true
I20260929 20:19:34.652629 9503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bce96c7a5293471f945fcf59f7343db2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:19:34.657297 9503 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bce96c7a5293471f945fcf59f7343db2 [term 1 FOLLOWER]: Refusing update from remote peer 7228480d03d7445989d1633de8d1c86f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:19:34.660636 9644 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce96c7a5293471f945fcf59f7343db2" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 37069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:19:34.660694 9649 sys_catalog.cc:455] T 00000000000000000000000000000000 P bce96c7a5293471f945fcf59f7343db2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57342817bee488a8f96e0d0d6b9b14f" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 38739 } } peers { permanent_uuid: "bce96c7a5293471f945fcf59f7343db2" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 37069 } } peers { permanent_uuid: "7228480d03d7445989d1633de8d1c86f" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 40453 } } }
I20260929 20:19:34.661718 9631 sys_catalog.cc:455] T 00000000000000000000000000000000 P d57342817bee488a8f96e0d0d6b9b14f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7228480d03d7445989d1633de8d1c86f. Latest consensus state: current_term: 1 leader_uuid: "7228480d03d7445989d1633de8d1c86f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57342817bee488a8f96e0d0d6b9b14f" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 38739 } } peers { permanent_uuid: "bce96c7a5293471f945fcf59f7343db2" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 37069 } } peers { permanent_uuid: "7228480d03d7445989d1633de8d1c86f" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 40453 } } }
I20260929 20:19:34.662060 9631 sys_catalog.cc:458] T 00000000000000000000000000000000 P d57342817bee488a8f96e0d0d6b9b14f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:19:34.661993 9649 sys_catalog.cc:458] T 00000000000000000000000000000000 P bce96c7a5293471f945fcf59f7343db2 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:19:34.673861 9617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7228480d03d7445989d1633de8d1c86f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57342817bee488a8f96e0d0d6b9b14f" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 38739 } } peers { permanent_uuid: "bce96c7a5293471f945fcf59f7343db2" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 37069 } } peers { permanent_uuid: "7228480d03d7445989d1633de8d1c86f" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 40453 } } }
I20260929 20:19:34.674245 9617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f [sys.catalog]: This master's current role is: LEADER
I20260929 20:19:34.674326 9647 catalog_manager.cc:1347] Generated new cluster ID: e7d3df035025410f8ae1be6a98d94cc8
I20260929 20:19:34.674414 9647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:19:34.678993 9631 sys_catalog.cc:455] T 00000000000000000000000000000000 P d57342817bee488a8f96e0d0d6b9b14f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7228480d03d7445989d1633de8d1c86f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57342817bee488a8f96e0d0d6b9b14f" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 38739 } } peers { permanent_uuid: "bce96c7a5293471f945fcf59f7343db2" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 37069 } } peers { permanent_uuid: "7228480d03d7445989d1633de8d1c86f" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 40453 } } }
I20260929 20:19:34.679407 9631 sys_catalog.cc:458] T 00000000000000000000000000000000 P d57342817bee488a8f96e0d0d6b9b14f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:19:34.689685 9647 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:19:34.691572 9647 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:19:34.707417 9649 sys_catalog.cc:455] T 00000000000000000000000000000000 P bce96c7a5293471f945fcf59f7343db2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7228480d03d7445989d1633de8d1c86f. Latest consensus state: current_term: 1 leader_uuid: "7228480d03d7445989d1633de8d1c86f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57342817bee488a8f96e0d0d6b9b14f" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 38739 } } peers { permanent_uuid: "bce96c7a5293471f945fcf59f7343db2" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 37069 } } peers { permanent_uuid: "7228480d03d7445989d1633de8d1c86f" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 40453 } } }
I20260929 20:19:34.707856 9649 sys_catalog.cc:458] T 00000000000000000000000000000000 P bce96c7a5293471f945fcf59f7343db2 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:19:34.709707 9644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7228480d03d7445989d1633de8d1c86f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57342817bee488a8f96e0d0d6b9b14f" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 38739 } } peers { permanent_uuid: "bce96c7a5293471f945fcf59f7343db2" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 37069 } } peers { permanent_uuid: "7228480d03d7445989d1633de8d1c86f" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 40453 } } }
I20260929 20:19:34.710064 9644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f [sys.catalog]: This master's current role is: LEADER
I20260929 20:19:34.712713 9656 mvcc.cc:205] Tried to move back new op lower bound from 7334761163296559104 to 7334761163090808832. Current Snapshot: MvccSnapshot[applied={T|T < 7334761163296559104}]
I20260929 20:19:34.721107 9647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f: Generated new TSK 0
I20260929 20:19:34.722169 9647 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:19:34.783183 9649 sys_catalog.cc:455] T 00000000000000000000000000000000 P bce96c7a5293471f945fcf59f7343db2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7228480d03d7445989d1633de8d1c86f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57342817bee488a8f96e0d0d6b9b14f" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 38739 } } peers { permanent_uuid: "bce96c7a5293471f945fcf59f7343db2" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 37069 } } peers { permanent_uuid: "7228480d03d7445989d1633de8d1c86f" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 40453 } } }
I20260929 20:19:34.783613 9649 sys_catalog.cc:458] T 00000000000000000000000000000000 P bce96c7a5293471f945fcf59f7343db2 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:19:34.839042 9672 catalog_manager.cc:1259] Loaded cluster ID: e7d3df035025410f8ae1be6a98d94cc8
I20260929 20:19:34.839234 9672 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bce96c7a5293471f945fcf59f7343db2: loading cluster ID for follower catalog manager: success
I20260929 20:19:34.843011 9672 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bce96c7a5293471f945fcf59f7343db2: acquiring CA information for follower catalog manager: success
I20260929 20:19:34.860275 9672 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bce96c7a5293471f945fcf59f7343db2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 20:19:35.509230 9642 catalog_manager.cc:1259] Loaded cluster ID: e7d3df035025410f8ae1be6a98d94cc8
I20260929 20:19:35.509354 9642 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d57342817bee488a8f96e0d0d6b9b14f: loading cluster ID for follower catalog manager: success
I20260929 20:19:35.512017 9642 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d57342817bee488a8f96e0d0d6b9b14f: acquiring CA information for follower catalog manager: success
I20260929 20:19:35.513899 9642 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d57342817bee488a8f96e0d0d6b9b14f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 20:19:35.870316 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 9389
I20260929 20:19:35.871627 9451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:19:35.979700 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 9389
I20260929 20:19:36.000985 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 9463
I20260929 20:19:36.002159 9525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:19:36.063159 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:36.062316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:38739 (local address 127.0.0.1:34254)
0929 20:19:36.062853 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:36.062984 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":281,"threads_started":1}
W20260929 20:19:36.063506 9555 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.185.126:38739: connect: Connection refused (error 111)
I20260929 20:19:36.064896 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:36.064560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:38739 (local address 127.0.0.1:34262)
0929 20:19:36.064709 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:36.064786 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 20:19:36.065076 9555 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7228480d03d7445989d1633de8d1c86f -> Peer d57342817bee488a8f96e0d0d6b9b14f (127.27.185.126:38739): Couldn't send request to peer d57342817bee488a8f96e0d0d6b9b14f. Status: Network error: Client connection negotiation failed: client connection to 127.27.185.126:38739: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 20:19:36.110560 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 9463
I20260929 20:19:36.131088 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 9537
I20260929 20:19:36.132321 9599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:19:36.241647 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 9537
2026-09-29T20:19:36Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4764 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:19:36 dist-test-slave-4cx3 krb5kdc[9681](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:19:36 dist-test-slave-4cx3 krb5kdc[9681](info): set up 2 sockets
Sep 29 20:19:36 dist-test-slave-4cx3 krb5kdc[9681](info): commencing operation
krb5kdc: starting...
W20260929 20:19:37.378733 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:19:37 dist-test-slave-4cx3 krb5kdc[9681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:19:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:19:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
I20260929 20:19:37.619092 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:41571
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:46077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:41571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:19:37.958905 9694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:19:37.959268 9694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:19:37.959492 9694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:19:37.966997 9694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:19:37.967164 9694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:19:37.967273 9694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:19:37.967409 9694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:19:37.967484 9694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:19:37.977823 9694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:46077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:41571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:41571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:19:37.979205 9694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:19:37.980647 9694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:19:37.992534 9699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:37.992635 9700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:37.992841 9702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:19:37.993844 9694 server_base.cc:1034] running on GCE node
I20260929 20:19:37.994661 9694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:19:37.996160 9694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:19:37.997392 9694 hybrid_clock.cc:648] HybridClock initialized: now 1790713177997353 us; error 62 us; skew 500 ppm
I20260929 20:19:37.997731 9694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:19:37 dist-test-slave-4cx3 krb5kdc[9681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713177, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:19:38.000921 9694 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:19:38.002463 9694 webserver.cc:460] Webserver started at http://127.27.185.126:36115/ using document root <none> and password file <none>
I20260929 20:19:38.003163 9694 fs_manager.cc:359] Metadata directory not provided
I20260929 20:19:38.003305 9694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:19:38.003654 9694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:19:38.006711 9694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "3f7f9f7155f0467ebec888f7af510f48"
format_stamp: "Formatted at 2026-09-29 20:19:38 on dist-test-slave-4cx3"
server_key: "5da27d2566d4df3c6c2370133ea4e1cc"
server_key_iv: "fc2553784b940b6e8bcdcd34289ab0fd"
server_key_version: "encryptionkey@0"
I20260929 20:19:38.007543 9694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "3f7f9f7155f0467ebec888f7af510f48"
format_stamp: "Formatted at 2026-09-29 20:19:38 on dist-test-slave-4cx3"
server_key: "5da27d2566d4df3c6c2370133ea4e1cc"
server_key_iv: "fc2553784b940b6e8bcdcd34289ab0fd"
server_key_version: "encryptionkey@0"
I20260929 20:19:38.012856 9694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 20:19:38.016405 9709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:38.017756 9694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:19:38.017938 9694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "3f7f9f7155f0467ebec888f7af510f48"
format_stamp: "Formatted at 2026-09-29 20:19:38 on dist-test-slave-4cx3"
server_key: "5da27d2566d4df3c6c2370133ea4e1cc"
server_key_iv: "fc2553784b940b6e8bcdcd34289ab0fd"
server_key_version: "encryptionkey@0"
I20260929 20:19:38.018142 9694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:19:38.042829 9694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:19:38.047143 9694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:19:38.047492 9694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:19:38.067022 9694 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:41571
I20260929 20:19:38.067041 9760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:41571 every 8 connection(s)
I20260929 20:19:38.068492 9694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:19:38.072228 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 9694
I20260929 20:19:38.072359 9761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:19:38.072449 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:19:38.073168 28389 external_mini_cluster.cc:1428] Setting key 7788570f4cfef51646095a39148ecbe6
I20260929 20:19:38.087138 9761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f7f9f7155f0467ebec888f7af510f48: Bootstrap starting.
Sep 29 20:19:38 dist-test-slave-4cx3 krb5kdc[9681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:19:38.090595 9761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f7f9f7155f0467ebec888f7af510f48: Neither blocks nor log segments found. Creating new log.
I20260929 20:19:38.092092 9761 log.cc:824] T 00000000000000000000000000000000 P 3f7f9f7155f0467ebec888f7af510f48: Log is configured to *not* fsync() on all Append() calls
I20260929 20:19:38.095402 9761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f7f9f7155f0467ebec888f7af510f48: No bootstrap required, opened a new log
I20260929 20:19:38.098114 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:38.075067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45498 (local address 127.27.185.126:41571)
0929 20:19:38.075972 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:38.075988 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:38.076048 (+ 60us) server_negotiation.cc:407] Connection header received
0929 20:19:38.077816 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:38.077851 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:38.078420 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:38.078900 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:38.080417 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:38.082362 (+ 1945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:38.084008 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:38.085176 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:38.085361 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:38.089656 (+ 4295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:38.089667 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:38.089689 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:38.089739 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:38.093512 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:38.094279 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:38.094287 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:38.094295 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:38.094494 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:38.094857 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:38.094862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:38.094865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:38.095602 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:38.095907 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:38.096262 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 20:19:38.096689 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":383,"threads_started":1}
I20260929 20:19:38.102236 28389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:19:38.104801 9761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f7f9f7155f0467ebec888f7af510f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f7f9f7155f0467ebec888f7af510f48" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41571 } }
I20260929 20:19:38.105237 9761 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f7f9f7155f0467ebec888f7af510f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:19:38.105365 9761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f7f9f7155f0467ebec888f7af510f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f7f9f7155f0467ebec888f7af510f48, State: Initialized, Role: FOLLOWER
I20260929 20:19:38.105959 9761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f7f9f7155f0467ebec888f7af510f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f7f9f7155f0467ebec888f7af510f48" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41571 } }
I20260929 20:19:38.106208 9761 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f7f9f7155f0467ebec888f7af510f48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:19:38.106346 9761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f7f9f7155f0467ebec888f7af510f48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:19:38.106509 9761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f7f9f7155f0467ebec888f7af510f48 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:19:38.108716 9761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f7f9f7155f0467ebec888f7af510f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f7f9f7155f0467ebec888f7af510f48" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41571 } }
I20260929 20:19:38.109195 9761 leader_election.cc:302] T 00000000000000000000000000000000 P 3f7f9f7155f0467ebec888f7af510f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f7f9f7155f0467ebec888f7af510f48; no voters:
I20260929 20:19:38.109768 9761 leader_election.cc:288] T 00000000000000000000000000000000 P 3f7f9f7155f0467ebec888f7af510f48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:19:38.109891 9766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f7f9f7155f0467ebec888f7af510f48 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:19:38.110610 9766 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f7f9f7155f0467ebec888f7af510f48 [term 1 LEADER]: Becoming Leader. State: Replica: 3f7f9f7155f0467ebec888f7af510f48, State: Running, Role: LEADER
I20260929 20:19:38.111195 9766 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f7f9f7155f0467ebec888f7af510f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f7f9f7155f0467ebec888f7af510f48" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41571 } }
I20260929 20:19:38.111829 9761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f7f9f7155f0467ebec888f7af510f48 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:19:38.113893 9768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f7f9f7155f0467ebec888f7af510f48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f7f9f7155f0467ebec888f7af510f48. Latest consensus state: current_term: 1 leader_uuid: "3f7f9f7155f0467ebec888f7af510f48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f7f9f7155f0467ebec888f7af510f48" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41571 } } }
I20260929 20:19:38.114099 9767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f7f9f7155f0467ebec888f7af510f48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f7f9f7155f0467ebec888f7af510f48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f7f9f7155f0467ebec888f7af510f48" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41571 } } }
I20260929 20:19:38.114306 9768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f7f9f7155f0467ebec888f7af510f48 [sys.catalog]: This master's current role is: LEADER
I20260929 20:19:38.114367 9767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f7f9f7155f0467ebec888f7af510f48 [sys.catalog]: This master's current role is: LEADER
I20260929 20:19:38.116179 9774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:19:38.121392 9774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:19:38.130484 9774 catalog_manager.cc:1347] Generated new cluster ID: ec6326feb00f46cabf5b496959e4a95b
I20260929 20:19:38.130599 9774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:19:38.140648 9774 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:19:38.141896 9774 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:19:38.148597 9774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f7f9f7155f0467ebec888f7af510f48: Generated new TSK 0
I20260929 20:19:38.149350 9774 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:19:38.162911 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 9694
I20260929 20:19:38.164167 9756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:19:38.253783 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 9694
2026-09-29T20:19:38Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2007 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:19:38 dist-test-slave-4cx3 krb5kdc[9789](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:19:38 dist-test-slave-4cx3 krb5kdc[9789](info): set up 2 sockets
Sep 29 20:19:38 dist-test-slave-4cx3 krb5kdc[9789](info): commencing operation
krb5kdc: starting...
W20260929 20:19:39.366940 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:19:39 dist-test-slave-4cx3 krb5kdc[9789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:19:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:19:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
I20260929 20:19:39.602869 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:33551
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:39637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:33551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:19:39.936633 9802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:19:39.936920 9802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:19:39.937021 9802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:19:39.944125 9802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:19:39.944231 9802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:19:39.944281 9802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:19:39.944339 9802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:19:39.944382 9802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:19:39.954000 9802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:39637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:33551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:33551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:19:39.955265 9802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:19:39.956564 9802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:19:39.969211 9807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:39.969281 9810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:39.969228 9808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:19:39.970269 9802 server_base.cc:1034] running on GCE node
I20260929 20:19:39.971104 9802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:19:39.972676 9802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:19:39.973917 9802 hybrid_clock.cc:648] HybridClock initialized: now 1790713179973868 us; error 58 us; skew 500 ppm
I20260929 20:19:39.974270 9802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:19:39 dist-test-slave-4cx3 krb5kdc[9789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713179, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:19:39.977450 9802 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:19:39.979012 9802 webserver.cc:460] Webserver started at http://127.27.185.126:46171/ using document root <none> and password file <none>
I20260929 20:19:39.979786 9802 fs_manager.cc:359] Metadata directory not provided
I20260929 20:19:39.979929 9802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:19:39.980238 9802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:19:39.983217 9802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "e1cb929ec49c473a8f68aedf49e45618"
format_stamp: "Formatted at 2026-09-29 20:19:39 on dist-test-slave-4cx3"
server_key: "ecf1ffcd6da3cef7286d6f7ebe647e06"
server_key_iv: "6691a4912ef4a787ca18e1155de1e399"
server_key_version: "encryptionkey@0"
I20260929 20:19:39.984061 9802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "e1cb929ec49c473a8f68aedf49e45618"
format_stamp: "Formatted at 2026-09-29 20:19:39 on dist-test-slave-4cx3"
server_key: "ecf1ffcd6da3cef7286d6f7ebe647e06"
server_key_iv: "6691a4912ef4a787ca18e1155de1e399"
server_key_version: "encryptionkey@0"
I20260929 20:19:39.989235 9802 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:19:39.992848 9817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:39.994153 9802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:19:39.994340 9802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "e1cb929ec49c473a8f68aedf49e45618"
format_stamp: "Formatted at 2026-09-29 20:19:39 on dist-test-slave-4cx3"
server_key: "ecf1ffcd6da3cef7286d6f7ebe647e06"
server_key_iv: "6691a4912ef4a787ca18e1155de1e399"
server_key_version: "encryptionkey@0"
I20260929 20:19:39.994544 9802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:19:40.014894 9802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:19:40.019598 9802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:19:40.019891 9802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:19:40.039811 9802 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:33551
I20260929 20:19:40.039834 9868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:33551 every 8 connection(s)
I20260929 20:19:40.041282 9802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:19:40.045105 9869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:19:40.045223 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 9802
I20260929 20:19:40.045456 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:19:40.046157 28389 external_mini_cluster.cc:1428] Setting key c6dbd5e74789e4dd02474554944e542c
I20260929 20:19:40.059337 9869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618: Bootstrap starting.
Sep 29 20:19:40 dist-test-slave-4cx3 krb5kdc[9789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:19:40.063081 9869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618: Neither blocks nor log segments found. Creating new log.
I20260929 20:19:40.064224 9869 log.cc:824] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618: Log is configured to *not* fsync() on all Append() calls
I20260929 20:19:40.067225 9869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618: No bootstrap required, opened a new log
I20260929 20:19:40.070240 9872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:40.048229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49208 (local address 127.27.185.126:33551)
0929 20:19:40.049028 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:40.049041 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:40.049084 (+ 43us) server_negotiation.cc:407] Connection header received
0929 20:19:40.051136 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:40.051160 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:40.051673 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:40.052031 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:40.053849 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:40.055131 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:40.057084 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:40.057807 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:40.057958 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:40.062812 (+ 4854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:40.062820 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:40.062835 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:40.062876 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:40.066014 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:40.066729 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:40.066735 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:40.066744 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:40.066907 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:40.067295 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:40.067299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:40.067301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:40.067942 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:40.068291 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:40.068730 (+ 439us) server_negotiation.cc:299] Negotiation successful
0929 20:19:40.069180 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":382,"threads_started":1}
I20260929 20:19:40.075562 9869 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cb929ec49c473a8f68aedf49e45618" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33551 } }
I20260929 20:19:40.075989 9869 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:19:40.076097 9869 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1cb929ec49c473a8f68aedf49e45618, State: Initialized, Role: FOLLOWER
I20260929 20:19:40.076710 9869 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cb929ec49c473a8f68aedf49e45618" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33551 } }
I20260929 20:19:40.076977 9869 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:19:40.077121 9869 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:19:40.077303 9869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:19:40.080027 9869 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cb929ec49c473a8f68aedf49e45618" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33551 } }
I20260929 20:19:40.080631 9869 leader_election.cc:302] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1cb929ec49c473a8f68aedf49e45618; no voters:
I20260929 20:19:40.081198 9869 leader_election.cc:288] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:19:40.081377 9874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:19:40.082095 9874 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [term 1 LEADER]: Becoming Leader. State: Replica: e1cb929ec49c473a8f68aedf49e45618, State: Running, Role: LEADER
I20260929 20:19:40.082655 9874 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cb929ec49c473a8f68aedf49e45618" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33551 } }
I20260929 20:19:40.083462 9869 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:19:40.085630 9876 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1cb929ec49c473a8f68aedf49e45618. Latest consensus state: current_term: 1 leader_uuid: "e1cb929ec49c473a8f68aedf49e45618" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cb929ec49c473a8f68aedf49e45618" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33551 } } }
I20260929 20:19:40.085970 9876 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [sys.catalog]: This master's current role is: LEADER
I20260929 20:19:40.086983 9881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:19:40.086802 9875 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1cb929ec49c473a8f68aedf49e45618" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cb929ec49c473a8f68aedf49e45618" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33551 } } }
I20260929 20:19:40.087199 9875 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [sys.catalog]: This master's current role is: LEADER
I20260929 20:19:40.094412 9881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:19:40.105103 9881 catalog_manager.cc:1347] Generated new cluster ID: 789f194adc19451ab36e64680873055b
I20260929 20:19:40.105248 9881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:19:40.118402 9881 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:19:40.120441 9881 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:19:40.134630 9881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618: Generated new TSK 0
I20260929 20:19:40.135982 9881 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
I20260929 20:19:40.193207 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:0
--local_ip_for_outbound_sockets=127.27.185.65
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:33551
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:39637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:19:40.527077 9895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:19:40.527350 9895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:19:40.527550 9895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:19:40.534843 9895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:19:40.534948 9895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:19:40.535113 9895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:19:40.544960 9895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:39637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:33551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:19:40.546274 9895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:19:40.547624 9895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:19:40.560758 9900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:40.560730 9901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:40.560803 9903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:19:40.561434 9895 server_base.cc:1034] running on GCE node
I20260929 20:19:40.562049 9895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:19:40.562937 9895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:19:40.564162 9895 hybrid_clock.cc:648] HybridClock initialized: now 1790713180564115 us; error 43 us; skew 500 ppm
I20260929 20:19:40.564472 9895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:19:40 dist-test-slave-4cx3 krb5kdc[9789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713180, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:19:40.567536 9895 init.cc:377] Logged in from keytab as kudu/127.27.185.65@KRBTEST.COM (short username kudu)
I20260929 20:19:40.568878 9895 webserver.cc:460] Webserver started at http://127.27.185.65:45195/ using document root <none> and password file <none>
I20260929 20:19:40.569556 9895 fs_manager.cc:359] Metadata directory not provided
I20260929 20:19:40.569697 9895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:19:40.570000 9895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:19:40.572930 9895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/data/instance:
uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f"
format_stamp: "Formatted at 2026-09-29 20:19:40 on dist-test-slave-4cx3"
server_key: "98de184ff94f09ec409c8855a63cce06"
server_key_iv: "5a76d19d0858b5bf110e7afe8bd1aec4"
server_key_version: "encryptionkey@0"
I20260929 20:19:40.573701 9895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance:
uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f"
format_stamp: "Formatted at 2026-09-29 20:19:40 on dist-test-slave-4cx3"
server_key: "98de184ff94f09ec409c8855a63cce06"
server_key_iv: "5a76d19d0858b5bf110e7afe8bd1aec4"
server_key_version: "encryptionkey@0"
I20260929 20:19:40.578753 9895 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 20:19:40.582186 9910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:40.583647 9895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:19:40.583811 9895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f"
format_stamp: "Formatted at 2026-09-29 20:19:40 on dist-test-slave-4cx3"
server_key: "98de184ff94f09ec409c8855a63cce06"
server_key_iv: "5a76d19d0858b5bf110e7afe8bd1aec4"
server_key_version: "encryptionkey@0"
I20260929 20:19:40.584002 9895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:19:40.610226 9895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:19:40.614243 9895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:19:40.614526 9895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:19:40.615540 9895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:19:40.616943 9895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:19:40.617053 9895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:40.617185 9895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:19:40.617252 9895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:40.650820 9895 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:33189
I20260929 20:19:40.650857 10022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:33189 every 8 connection(s)
I20260929 20:19:40.652191 9895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:19:40.656642 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 9895
I20260929 20:19:40.656896 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance
I20260929 20:19:40.657771 28389 external_mini_cluster.cc:1428] Setting key b2f43265d36523c66ab6a27f8c16e42c
Sep 29 20:19:40 dist-test-slave-4cx3 krb5kdc[9789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713180, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:19:40.675463 10025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:40.654872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:49169 (local address 127.27.185.126:33551)
0929 20:19:40.655348 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:40.655354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:40.656303 (+ 949us) server_negotiation.cc:407] Connection header received
0929 20:19:40.658931 (+ 2628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:40.658939 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:40.659013 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:40.659110 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:40.661230 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:40.661974 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:40.664619 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:40.665040 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:40.665145 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:40.669557 (+ 4412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:40.669563 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:40.669566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:40.669626 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:40.672251 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:40.673043 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:40.673047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:40.673050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:40.673170 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:40.673612 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:40.673617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:40.673620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:40.673897 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:40.674063 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:40.674805 (+ 742us) server_negotiation.cc:299] Negotiation successful
0929 20:19:40.675051 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":258,"threads_started":1}
I20260929 20:19:40.676522 10026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:40.655308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.65:49169)
0929 20:19:40.656075 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:40.656162 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:40.658663 (+ 2501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:40.659442 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:40.659454 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:40.660159 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:19:40.661036 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:40.661064 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:40.662149 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:40.662158 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:40.664443 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:40.664469 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:40.665304 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:40.665314 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:40.665692 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:40.666467 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:19:40.666502 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:19:40.669275 (+ 2773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:19:40.672408 (+ 3133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:40.672418 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:40.672437 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:40.672902 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:40.673281 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:40.673286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:40.673290 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:40.673490 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:40.674028 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:19:40.674036 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:19:40.674410 (+ 374us) client_negotiation.cc:769] Sending connection context
0929 20:19:40.674690 (+ 280us) client_negotiation.cc:240] Negotiation successful
0929 20:19:40.675270 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":269,"threads_started":1}
I20260929 20:19:40.678337 10023 heartbeater.cc:347] Connected to a master server at 127.27.185.126:33551
I20260929 20:19:40.678683 10023 heartbeater.cc:464] Registering TS with master...
I20260929 20:19:40.679495 10023 heartbeater.cc:511] Master 127.27.185.126:33551 requested a full tablet report, sending...
I20260929 20:19:40.681681 9834 ts_manager.cc:194] Registered new tserver with Master: 9c0fe02fa1b14d13aa929b74e09d1f7f (127.27.185.65:33189)
WARNING: no policy specified for kudu/127.27.185.66@KRBTEST.COM; defaulting to no policy
I20260929 20:19:40.683171 9834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.65@KRBTEST.COM'} at 127.27.185.65:49169
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
I20260929 20:19:40.714439 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:0
--local_ip_for_outbound_sockets=127.27.185.66
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:33551
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:39637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:19:41.041703 10029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:19:41.041965 10029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:19:41.042057 10029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:19:41.049489 10029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:19:41.049592 10029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:19:41.049762 10029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:19:41.059666 10029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:39637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:33551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:19:41.060986 10029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:19:41.062289 10029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:19:41.076095 10035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:41.076129 10034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:41.076198 10037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:19:41.076907 10029 server_base.cc:1034] running on GCE node
I20260929 20:19:41.077544 10029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:19:41.078469 10029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:19:41.079664 10029 hybrid_clock.cc:648] HybridClock initialized: now 1790713181079616 us; error 35 us; skew 500 ppm
I20260929 20:19:41.079998 10029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:19:41 dist-test-slave-4cx3 krb5kdc[9789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713181, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:19:41.083114 10029 init.cc:377] Logged in from keytab as kudu/127.27.185.66@KRBTEST.COM (short username kudu)
I20260929 20:19:41.084904 10029 webserver.cc:460] Webserver started at http://127.27.185.66:45403/ using document root <none> and password file <none>
I20260929 20:19:41.085603 10029 fs_manager.cc:359] Metadata directory not provided
I20260929 20:19:41.085764 10029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:19:41.086068 10029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:19:41.089063 10029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/data/instance:
uuid: "693d5229f3094dd9896297d9ace72929"
format_stamp: "Formatted at 2026-09-29 20:19:41 on dist-test-slave-4cx3"
server_key: "4066633c4519859c978d89e6e6143178"
server_key_iv: "ef5f32b5229503a03de2cda201f3e8a5"
server_key_version: "encryptionkey@0"
I20260929 20:19:41.089898 10029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance:
uuid: "693d5229f3094dd9896297d9ace72929"
format_stamp: "Formatted at 2026-09-29 20:19:41 on dist-test-slave-4cx3"
server_key: "4066633c4519859c978d89e6e6143178"
server_key_iv: "ef5f32b5229503a03de2cda201f3e8a5"
server_key_version: "encryptionkey@0"
I20260929 20:19:41.095268 10029 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:19:41.098803 10044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:41.100299 10029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:19:41.100484 10029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "693d5229f3094dd9896297d9ace72929"
format_stamp: "Formatted at 2026-09-29 20:19:41 on dist-test-slave-4cx3"
server_key: "4066633c4519859c978d89e6e6143178"
server_key_iv: "ef5f32b5229503a03de2cda201f3e8a5"
server_key_version: "encryptionkey@0"
I20260929 20:19:41.100669 10029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:19:41.123665 10029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:19:41.128090 10029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:19:41.128386 10029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:19:41.129382 10029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:19:41.130759 10029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:19:41.130841 10029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:41.131006 10029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:19:41.131055 10029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:41.164721 10029 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:45885
I20260929 20:19:41.164767 10156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:45885 every 8 connection(s)
I20260929 20:19:41.166205 10029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:19:41.167587 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 10029
I20260929 20:19:41.167771 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance
I20260929 20:19:41.168360 28389 external_mini_cluster.cc:1428] Setting key 6a4c49166f33afb6bda7a3cccc3e1b52
Sep 29 20:19:41 dist-test-slave-4cx3 krb5kdc[9789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713181, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:19:41.188789 10025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:41.169011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:42579 (local address 127.27.185.126:33551)
0929 20:19:41.169207 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:41.169212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:41.170582 (+ 1370us) server_negotiation.cc:407] Connection header received
0929 20:19:41.172468 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:41.172473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:41.172560 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:41.172660 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:41.174132 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.174831 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.176443 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.177182 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.177324 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:41.181200 (+ 3876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:41.181209 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:41.181217 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:41.181263 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:41.185064 (+ 3801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:41.185880 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:41.185889 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:41.185895 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:41.186052 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:41.186538 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:41.186546 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:41.186552 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:41.186943 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:41.187107 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:41.188021 (+ 914us) server_negotiation.cc:299] Negotiation successful
0929 20:19:41.188292 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:19:41.189641 10159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:41.169600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.66:42579)
0929 20:19:41.170426 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:41.170472 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:41.172199 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:41.172886 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:41.172895 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:41.173385 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:19:41.173966 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:41.173983 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.174977 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.174988 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:41.176294 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:41.176306 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.177414 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.177419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:41.177633 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:41.178413 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:19:41.178437 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:19:41.180954 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:19:41.185202 (+ 4248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:41.185209 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:41.185223 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:41.185741 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:41.186184 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:41.186188 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:41.186190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:41.186405 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:41.187176 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:19:41.187182 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:19:41.187570 (+ 388us) client_negotiation.cc:769] Sending connection context
0929 20:19:41.187879 (+ 309us) client_negotiation.cc:240] Negotiation successful
0929 20:19:41.188416 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":302,"threads_started":1}
I20260929 20:19:41.191200 10157 heartbeater.cc:347] Connected to a master server at 127.27.185.126:33551
I20260929 20:19:41.191620 10157 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.27.185.67@KRBTEST.COM; defaulting to no policy
I20260929 20:19:41.192628 10157 heartbeater.cc:511] Master 127.27.185.126:33551 requested a full tablet report, sending...
I20260929 20:19:41.194176 9834 ts_manager.cc:194] Registered new tserver with Master: 693d5229f3094dd9896297d9ace72929 (127.27.185.66:45885)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.67@KRBTEST.COM" created.
I20260929 20:19:41.195149 9834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.66@KRBTEST.COM'} at 127.27.185.66:42579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
I20260929 20:19:41.223796 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:0
--local_ip_for_outbound_sockets=127.27.185.67
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:33551
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:39637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:19:41.542640 10162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:19:41.542912 10162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:19:41.543078 10162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:19:41.550068 10162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:19:41.550163 10162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:19:41.550343 10162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:19:41.559558 10162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:39637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:33551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:19:41.560755 10162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:19:41.562016 10162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:19:41.575637 10167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:41.575479 10168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:41.575451 10170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:19:41.576752 10162 server_base.cc:1034] running on GCE node
I20260929 20:19:41.577373 10162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:19:41.578327 10162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:19:41.579548 10162 hybrid_clock.cc:648] HybridClock initialized: now 1790713181579502 us; error 36 us; skew 500 ppm
I20260929 20:19:41.579867 10162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:19:41 dist-test-slave-4cx3 krb5kdc[9789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713181, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:19:41.582940 10162 init.cc:377] Logged in from keytab as kudu/127.27.185.67@KRBTEST.COM (short username kudu)
I20260929 20:19:41.584578 10162 webserver.cc:460] Webserver started at http://127.27.185.67:41159/ using document root <none> and password file <none>
I20260929 20:19:41.585291 10162 fs_manager.cc:359] Metadata directory not provided
I20260929 20:19:41.585392 10162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:19:41.585639 10162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:19:41.588683 10162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/data/instance:
uuid: "6b2231cba4a6403d98b27afdb04948af"
format_stamp: "Formatted at 2026-09-29 20:19:41 on dist-test-slave-4cx3"
server_key: "582f80d8ec070c77a1e67cbda5ac14f8"
server_key_iv: "a84718b5c4e35b47529a28591586a599"
server_key_version: "encryptionkey@0"
I20260929 20:19:41.589450 10162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance:
uuid: "6b2231cba4a6403d98b27afdb04948af"
format_stamp: "Formatted at 2026-09-29 20:19:41 on dist-test-slave-4cx3"
server_key: "582f80d8ec070c77a1e67cbda5ac14f8"
server_key_iv: "a84718b5c4e35b47529a28591586a599"
server_key_version: "encryptionkey@0"
I20260929 20:19:41.594480 10162 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:19:41.597904 10177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:41.599327 10162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:19:41.599509 10162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "6b2231cba4a6403d98b27afdb04948af"
format_stamp: "Formatted at 2026-09-29 20:19:41 on dist-test-slave-4cx3"
server_key: "582f80d8ec070c77a1e67cbda5ac14f8"
server_key_iv: "a84718b5c4e35b47529a28591586a599"
server_key_version: "encryptionkey@0"
I20260929 20:19:41.599684 10162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:19:41.619621 10162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:19:41.623768 10162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:19:41.624004 10162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:19:41.624928 10162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:19:41.626209 10162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:19:41.626277 10162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:41.626360 10162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:19:41.626403 10162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:41.660430 10162 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:42645
I20260929 20:19:41.660552 10289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:42645 every 8 connection(s)
I20260929 20:19:41.661684 10162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:19:41.666261 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 10162
I20260929 20:19:41.666460 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance
I20260929 20:19:41.666916 28389 external_mini_cluster.cc:1428] Setting key 7205aaf2c62d265d8bcc56978f863ed2
Sep 29 20:19:41 dist-test-slave-4cx3 krb5kdc[9789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713181, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:19:41.682467 10025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:41.664410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:33845 (local address 127.27.185.126:33551)
0929 20:19:41.664586 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:41.664590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:41.665751 (+ 1161us) server_negotiation.cc:407] Connection header received
0929 20:19:41.667665 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:41.667670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:41.667768 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:41.667899 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:41.669859 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.670661 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.672273 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.672711 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.672809 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:41.676533 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:41.676538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:41.676541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:41.676573 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:41.679188 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:41.679971 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:41.679976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:41.679978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:41.680108 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:41.680551 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:41.680553 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:41.680555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:41.680885 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:41.681038 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:41.681900 (+ 862us) server_negotiation.cc:299] Negotiation successful
0929 20:19:41.682129 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 20:19:41.683478 10292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:41.664880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.67:33845)
0929 20:19:41.665580 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:41.665627 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:41.667318 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:41.668164 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:41.668173 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:41.668867 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:19:41.669668 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:41.669694 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.670814 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.670818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:41.672100 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:41.672111 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.672893 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.672897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:41.673082 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:41.673700 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:19:41.673725 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:19:41.676344 (+ 2619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:19:41.679297 (+ 2953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:41.679303 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:41.679314 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:41.679765 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:41.680200 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:41.680203 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:41.680205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:41.680434 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:41.681031 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:19:41.681039 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:19:41.681440 (+ 401us) client_negotiation.cc:769] Sending connection context
0929 20:19:41.681774 (+ 334us) client_negotiation.cc:240] Negotiation successful
0929 20:19:41.682283 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":267,"threads_started":1}
I20260929 20:19:41.684777 10290 heartbeater.cc:347] Connected to a master server at 127.27.185.126:33551
I20260929 20:19:41.685066 10290 heartbeater.cc:464] Registering TS with master...
I20260929 20:19:41.685806 10290 heartbeater.cc:511] Master 127.27.185.126:33551 requested a full tablet report, sending...
I20260929 20:19:41.686509 10023 heartbeater.cc:503] Master 127.27.185.126:33551 was elected leader, sending a full tablet report...
I20260929 20:19:41.687263 9834 ts_manager.cc:194] Registered new tserver with Master: 6b2231cba4a6403d98b27afdb04948af (127.27.185.67:42645)
I20260929 20:19:41.688258 9834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.67@KRBTEST.COM'} at 127.27.185.67:33845
I20260929 20:19:41.694545 28389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:19:41.711442 10025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:41.699083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49224 (local address 127.27.185.126:33551)
0929 20:19:41.699241 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:41.699245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:41.699732 (+ 487us) server_negotiation.cc:407] Connection header received
0929 20:19:41.699870 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:41.699874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:41.699937 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:41.700018 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:41.701379 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.702099 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.703598 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.704202 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.704296 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:41.706672 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:41.706677 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:41.706695 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:41.706736 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:41.709299 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:41.709889 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:41.709893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:41.709895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:41.710015 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:41.710304 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:41.710307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:41.710309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:41.710603 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:41.710782 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:41.711029 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 20:19:41.711179 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:19:41.715068 9834 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:19:41.721971 9834 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:19:41.758203 10305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:41.734306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34172 (local address 127.27.185.65:33189)
0929 20:19:41.746620 (+ 12314us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:41.746634 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:41.747704 (+ 1070us) server_negotiation.cc:407] Connection header received
0929 20:19:41.748135 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:41.748143 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:41.748415 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:41.748600 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:19:41.749957 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.751568 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.755019 (+ 3451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.756597 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.756771 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:41.757046 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:41.757713 (+ 667us) server_negotiation.cc:299] Negotiation successful
0929 20:19:41.757870 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12038,"thread_start_us":12313,"threads_started":1}
I20260929 20:19:41.758244 10306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:41.748058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:42645 (local address 127.0.0.1:55966)
0929 20:19:41.749574 (+ 1516us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:41.749593 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:41.749713 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:41.750859 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:41.750864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:41.750889 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:19:41.751252 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:41.751262 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.752855 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.752861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:41.755201 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:41.755216 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.757193 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.757200 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:41.757423 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:41.757468 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 20:19:41.757558 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 20:19:41.757764 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1371,"mutex_wait_us":29,"thread_start_us":1212,"threads_started":1}
I20260929 20:19:41.758229 10302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:41.733741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36772 (local address 127.27.185.66:45885)
0929 20:19:41.746710 (+ 12969us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:41.746723 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:41.747705 (+ 982us) server_negotiation.cc:407] Connection header received
0929 20:19:41.748132 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:41.748143 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:41.748432 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:41.748635 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:19:41.750395 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.751852 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.754977 (+ 3125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.756313 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.756564 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:41.756789 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:41.757651 (+ 862us) server_negotiation.cc:299] Negotiation successful
0929 20:19:41.757854 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12735,"thread_start_us":13006,"threads_started":1}
I20260929 20:19:41.758813 10303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:41.733752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:45885 (local address 127.0.0.1:36772)
0929 20:19:41.747539 (+ 13787us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:41.747577 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:41.747875 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:41.749004 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:41.749012 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:41.749686 (+ 674us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:19:41.750177 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:41.750191 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.751989 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.751994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:41.754793 (+ 2799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:41.754808 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.757034 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.757046 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:41.757291 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:41.757411 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 20:19:41.758238 (+ 827us) client_negotiation.cc:240] Negotiation successful
0929 20:19:41.758390 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13545,"mutex_wait_us":299,"spinlock_wait_cycles":1432960,"thread_start_us":13311,"threads_started":1}
I20260929 20:19:41.759467 10304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:41.733726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:33189 (local address 127.0.0.1:34172)
0929 20:19:41.747537 (+ 13811us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:41.747577 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:41.747874 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:41.748995 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:41.749006 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:41.749099 (+ 93us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:19:41.749740 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:41.749756 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.751678 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.751686 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:41.754795 (+ 3109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:41.754808 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.757205 (+ 2397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.757210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:41.757467 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:41.757492 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 20:19:41.757562 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 20:19:41.759091 (+ 1529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13502,"mutex_wait_us":1426,"spinlock_wait_cycles":26240,"thread_start_us":13412,"threads_started":1}
I20260929 20:19:41.759610 10307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:41.748371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55966 (local address 127.27.185.67:42645)
0929 20:19:41.750284 (+ 1913us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:41.750293 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:41.750327 (+ 34us) server_negotiation.cc:407] Connection header received
0929 20:19:41.750418 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:41.750425 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:41.750675 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:41.750862 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:19:41.751447 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.752687 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.755374 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.756903 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.758570 (+ 1667us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:41.758831 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:41.758909 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 20:19:41.759097 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1727,"thread_start_us":1680,"threads_started":1}
I20260929 20:19:41.763345 10225 tablet_service.cc:1467] Processing CreateTablet for tablet 1f696f1d1200466ab391debd7d67b20d (DEFAULT_TABLE table=test-table [id=c214059c648a43ea9ae9adcc08fc0af0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:19:41.763355 10092 tablet_service.cc:1467] Processing CreateTablet for tablet 1f696f1d1200466ab391debd7d67b20d (DEFAULT_TABLE table=test-table [id=c214059c648a43ea9ae9adcc08fc0af0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:19:41.764993 10225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f696f1d1200466ab391debd7d67b20d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:19:41.764995 10092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f696f1d1200466ab391debd7d67b20d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:19:41.767680 9958 tablet_service.cc:1467] Processing CreateTablet for tablet 1f696f1d1200466ab391debd7d67b20d (DEFAULT_TABLE table=test-table [id=c214059c648a43ea9ae9adcc08fc0af0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:19:41.768975 9958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f696f1d1200466ab391debd7d67b20d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:19:41.781077 10308 tablet_bootstrap.cc:492] T 1f696f1d1200466ab391debd7d67b20d P 9c0fe02fa1b14d13aa929b74e09d1f7f: Bootstrap starting.
I20260929 20:19:41.783689 10309 tablet_bootstrap.cc:492] T 1f696f1d1200466ab391debd7d67b20d P 6b2231cba4a6403d98b27afdb04948af: Bootstrap starting.
I20260929 20:19:41.784235 10310 tablet_bootstrap.cc:492] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929: Bootstrap starting.
I20260929 20:19:41.785219 10308 tablet_bootstrap.cc:654] T 1f696f1d1200466ab391debd7d67b20d P 9c0fe02fa1b14d13aa929b74e09d1f7f: Neither blocks nor log segments found. Creating new log.
I20260929 20:19:41.786549 10308 log.cc:824] T 1f696f1d1200466ab391debd7d67b20d P 9c0fe02fa1b14d13aa929b74e09d1f7f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:19:41.787293 10309 tablet_bootstrap.cc:654] T 1f696f1d1200466ab391debd7d67b20d P 6b2231cba4a6403d98b27afdb04948af: Neither blocks nor log segments found. Creating new log.
I20260929 20:19:41.787508 10310 tablet_bootstrap.cc:654] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929: Neither blocks nor log segments found. Creating new log.
I20260929 20:19:41.788599 10309 log.cc:824] T 1f696f1d1200466ab391debd7d67b20d P 6b2231cba4a6403d98b27afdb04948af: Log is configured to *not* fsync() on all Append() calls
I20260929 20:19:41.788995 10310 log.cc:824] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929: Log is configured to *not* fsync() on all Append() calls
I20260929 20:19:41.790534 10308 tablet_bootstrap.cc:492] T 1f696f1d1200466ab391debd7d67b20d P 9c0fe02fa1b14d13aa929b74e09d1f7f: No bootstrap required, opened a new log
I20260929 20:19:41.790809 10308 ts_tablet_manager.cc:1397] T 1f696f1d1200466ab391debd7d67b20d P 9c0fe02fa1b14d13aa929b74e09d1f7f: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 20:19:41.792047 10309 tablet_bootstrap.cc:492] T 1f696f1d1200466ab391debd7d67b20d P 6b2231cba4a6403d98b27afdb04948af: No bootstrap required, opened a new log
I20260929 20:19:41.792291 10310 tablet_bootstrap.cc:492] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929: No bootstrap required, opened a new log
I20260929 20:19:41.792353 10309 ts_tablet_manager.cc:1397] T 1f696f1d1200466ab391debd7d67b20d P 6b2231cba4a6403d98b27afdb04948af: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 20:19:41.792605 10310 ts_tablet_manager.cc:1397] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 20:19:41.798885 10308 raft_consensus.cc:348] T 1f696f1d1200466ab391debd7d67b20d P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "693d5229f3094dd9896297d9ace72929" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 45885 } } peers { permanent_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 33189 } } peers { permanent_uuid: "6b2231cba4a6403d98b27afdb04948af" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 42645 } }
I20260929 20:19:41.799887 10308 raft_consensus.cc:374] T 1f696f1d1200466ab391debd7d67b20d P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:19:41.800055 10308 raft_consensus.cc:729] T 1f696f1d1200466ab391debd7d67b20d P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c0fe02fa1b14d13aa929b74e09d1f7f, State: Initialized, Role: FOLLOWER
I20260929 20:19:41.800686 10308 consensus_queue.cc:260] T 1f696f1d1200466ab391debd7d67b20d P 9c0fe02fa1b14d13aa929b74e09d1f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "693d5229f3094dd9896297d9ace72929" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 45885 } } peers { permanent_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 33189 } } peers { permanent_uuid: "6b2231cba4a6403d98b27afdb04948af" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 42645 } }
I20260929 20:19:41.804736 10309 raft_consensus.cc:348] T 1f696f1d1200466ab391debd7d67b20d P 6b2231cba4a6403d98b27afdb04948af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "693d5229f3094dd9896297d9ace72929" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 45885 } } peers { permanent_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 33189 } } peers { permanent_uuid: "6b2231cba4a6403d98b27afdb04948af" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 42645 } }
I20260929 20:19:41.804740 10310 raft_consensus.cc:348] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "693d5229f3094dd9896297d9ace72929" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 45885 } } peers { permanent_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 33189 } } peers { permanent_uuid: "6b2231cba4a6403d98b27afdb04948af" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 42645 } }
I20260929 20:19:41.805217 10310 raft_consensus.cc:374] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:19:41.805217 10309 raft_consensus.cc:374] T 1f696f1d1200466ab391debd7d67b20d P 6b2231cba4a6403d98b27afdb04948af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:19:41.805330 10310 raft_consensus.cc:729] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 693d5229f3094dd9896297d9ace72929, State: Initialized, Role: FOLLOWER
I20260929 20:19:41.805343 10309 raft_consensus.cc:729] T 1f696f1d1200466ab391debd7d67b20d P 6b2231cba4a6403d98b27afdb04948af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b2231cba4a6403d98b27afdb04948af, State: Initialized, Role: FOLLOWER
I20260929 20:19:41.806005 10310 consensus_queue.cc:260] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "693d5229f3094dd9896297d9ace72929" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 45885 } } peers { permanent_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 33189 } } peers { permanent_uuid: "6b2231cba4a6403d98b27afdb04948af" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 42645 } }
I20260929 20:19:41.806010 10309 consensus_queue.cc:260] T 1f696f1d1200466ab391debd7d67b20d P 6b2231cba4a6403d98b27afdb04948af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "693d5229f3094dd9896297d9ace72929" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 45885 } } peers { permanent_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 33189 } } peers { permanent_uuid: "6b2231cba4a6403d98b27afdb04948af" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 42645 } }
I20260929 20:19:41.806350 10308 ts_tablet_manager.cc:1428] T 1f696f1d1200466ab391debd7d67b20d P 9c0fe02fa1b14d13aa929b74e09d1f7f: Time spent starting tablet: real 0.015s user 0.003s sys 0.010s
I20260929 20:19:41.808130 10309 ts_tablet_manager.cc:1428] T 1f696f1d1200466ab391debd7d67b20d P 6b2231cba4a6403d98b27afdb04948af: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260929 20:19:41.808600 10310 ts_tablet_manager.cc:1428] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
I20260929 20:19:41.808666 10290 heartbeater.cc:503] Master 127.27.185.126:33551 was elected leader, sending a full tablet report...
I20260929 20:19:41.808725 10157 heartbeater.cc:503] Master 127.27.185.126:33551 was elected leader, sending a full tablet report...
I20260929 20:19:41.830386 10315 raft_consensus.cc:482] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:19:41.830688 10315 raft_consensus.cc:504] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "693d5229f3094dd9896297d9ace72929" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 45885 } } peers { permanent_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 33189 } } peers { permanent_uuid: "6b2231cba4a6403d98b27afdb04948af" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 42645 } }
I20260929 20:19:41.832387 10315 leader_election.cc:288] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c0fe02fa1b14d13aa929b74e09d1f7f (127.27.185.65:33189), 6b2231cba4a6403d98b27afdb04948af (127.27.185.67:42645)
I20260929 20:19:41.839334 10305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:41.832726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:60927 (local address 127.27.185.65:33189)
0929 20:19:41.832964 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:41.832975 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:41.833536 (+ 561us) server_negotiation.cc:407] Connection header received
0929 20:19:41.833810 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:41.833816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:41.833918 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:41.833971 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:19:41.834860 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.835704 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.837582 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.838375 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.838489 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:41.838547 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:41.838931 (+ 384us) server_negotiation.cc:299] Negotiation successful
0929 20:19:41.839079 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:19:41.839439 10307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:41.833641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:41683 (local address 127.27.185.67:42645)
0929 20:19:41.833836 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:41.833842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:41.834180 (+ 338us) server_negotiation.cc:407] Connection header received
0929 20:19:41.834314 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:41.834321 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:41.834413 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:41.834505 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:19:41.835105 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.835996 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.837695 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.838481 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.838649 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:41.838720 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:41.839046 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 20:19:41.839184 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:19:41.839505 10317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:41.832702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:33189 (local address 127.27.185.66:60927)
0929 20:19:41.833398 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:41.833423 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:41.833614 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:41.834159 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:41.834165 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:41.834270 (+ 105us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:19:41.834701 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:41.834715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.835833 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.835839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:41.837420 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:41.837431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.838591 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.838597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:41.838776 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:41.838802 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 20:19:41.838877 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 20:19:41.838962 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":2,"spinlock_wait_cycles":23936,"thread_start_us":390,"threads_started":1}
I20260929 20:19:41.839676 10318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:41.833479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:42645 (local address 127.27.185.66:41683)
0929 20:19:41.834078 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:41.834091 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:41.834194 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:41.834536 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:41.834538 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:41.834555 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:19:41.834944 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:41.834958 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.836128 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.836131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:41.837541 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:41.837556 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.838692 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.838696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:41.838883 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:41.838916 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:19:41.839004 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 20:19:41.839129 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":309,"threads_started":1}
I20260929 20:19:41.840641 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f696f1d1200466ab391debd7d67b20d" candidate_uuid: "693d5229f3094dd9896297d9ace72929" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f" is_pre_election: true
I20260929 20:19:41.840641 10245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f696f1d1200466ab391debd7d67b20d" candidate_uuid: "693d5229f3094dd9896297d9ace72929" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b2231cba4a6403d98b27afdb04948af" is_pre_election: true
I20260929 20:19:41.841224 9978 raft_consensus.cc:2454] T 1f696f1d1200466ab391debd7d67b20d P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 693d5229f3094dd9896297d9ace72929 in term 0.
I20260929 20:19:41.841220 10245 raft_consensus.cc:2454] T 1f696f1d1200466ab391debd7d67b20d P 6b2231cba4a6403d98b27afdb04948af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 693d5229f3094dd9896297d9ace72929 in term 0.
I20260929 20:19:41.841881 10045 leader_election.cc:302] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 693d5229f3094dd9896297d9ace72929, 9c0fe02fa1b14d13aa929b74e09d1f7f; no voters:
I20260929 20:19:41.842212 10315 raft_consensus.cc:2781] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:19:41.842357 10315 raft_consensus.cc:482] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:19:41.842460 10315 raft_consensus.cc:3037] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:19:41.844676 10315 raft_consensus.cc:504] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "693d5229f3094dd9896297d9ace72929" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 45885 } } peers { permanent_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 33189 } } peers { permanent_uuid: "6b2231cba4a6403d98b27afdb04948af" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 42645 } }
I20260929 20:19:41.845233 10315 leader_election.cc:288] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929 [CANDIDATE]: Term 1 election: Requested vote from peers 9c0fe02fa1b14d13aa929b74e09d1f7f (127.27.185.65:33189), 6b2231cba4a6403d98b27afdb04948af (127.27.185.67:42645)
I20260929 20:19:41.845638 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f696f1d1200466ab391debd7d67b20d" candidate_uuid: "693d5229f3094dd9896297d9ace72929" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f"
I20260929 20:19:41.845731 10245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f696f1d1200466ab391debd7d67b20d" candidate_uuid: "693d5229f3094dd9896297d9ace72929" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b2231cba4a6403d98b27afdb04948af"
I20260929 20:19:41.845824 9978 raft_consensus.cc:3037] T 1f696f1d1200466ab391debd7d67b20d P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:19:41.845963 10245 raft_consensus.cc:3037] T 1f696f1d1200466ab391debd7d67b20d P 6b2231cba4a6403d98b27afdb04948af [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:19:41.848367 9978 raft_consensus.cc:2454] T 1f696f1d1200466ab391debd7d67b20d P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 693d5229f3094dd9896297d9ace72929 in term 1.
I20260929 20:19:41.848701 10245 raft_consensus.cc:2454] T 1f696f1d1200466ab391debd7d67b20d P 6b2231cba4a6403d98b27afdb04948af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 693d5229f3094dd9896297d9ace72929 in term 1.
I20260929 20:19:41.848841 10045 leader_election.cc:302] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 693d5229f3094dd9896297d9ace72929, 9c0fe02fa1b14d13aa929b74e09d1f7f; no voters:
I20260929 20:19:41.849246 10315 raft_consensus.cc:2781] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:19:41.849640 10315 raft_consensus.cc:686] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929 [term 1 LEADER]: Becoming Leader. State: Replica: 693d5229f3094dd9896297d9ace72929, State: Running, Role: LEADER
I20260929 20:19:41.850088 10315 consensus_queue.cc:237] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "693d5229f3094dd9896297d9ace72929" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 45885 } } peers { permanent_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 33189 } } peers { permanent_uuid: "6b2231cba4a6403d98b27afdb04948af" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 42645 } }
I20260929 20:19:41.854795 9832 catalog_manager.cc:5495] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929 reported cstate change: term changed from 0 to 1, leader changed from <none> to 693d5229f3094dd9896297d9ace72929 (127.27.185.66). New cstate: current_term: 1 leader_uuid: "693d5229f3094dd9896297d9ace72929" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "693d5229f3094dd9896297d9ace72929" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 45885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 33189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b2231cba4a6403d98b27afdb04948af" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 42645 } health_report { overall_health: UNKNOWN } } }
I20260929 20:19:41.884330 10302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:41.879013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36782 (local address 127.27.185.66:45885)
0929 20:19:41.879211 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:41.879215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:41.879234 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:19:41.879480 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:41.879484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:41.879568 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:41.879690 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:19:41.880050 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.880739 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.882179 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.882557 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.882646 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:41.883090 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:19:41.883698 (+ 608us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:19:41.883805 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:41.883933 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 20:19:41.884009 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:19:41.892568 9978 raft_consensus.cc:1261] T 1f696f1d1200466ab391debd7d67b20d P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 1 FOLLOWER]: Refusing update from remote peer 693d5229f3094dd9896297d9ace72929: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:19:41.892581 10245 raft_consensus.cc:1261] T 1f696f1d1200466ab391debd7d67b20d P 6b2231cba4a6403d98b27afdb04948af [term 1 FOLLOWER]: Refusing update from remote peer 693d5229f3094dd9896297d9ace72929: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:19:41.893522 10319 consensus_queue.cc:1035] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 33189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:19:41.893854 10315 consensus_queue.cc:1035] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b2231cba4a6403d98b27afdb04948af" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 42645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 20:19:41.905791 10024 tablet.cc:2369] T 1f696f1d1200466ab391debd7d67b20d P 9c0fe02fa1b14d13aa929b74e09d1f7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:19:41.908216 10323 mvcc.cc:205] Tried to move back new op lower bound from 7334761193021587456 to 7334761192861581312. Current Snapshot: MvccSnapshot[applied={T|T < 7334761193021587456}]
I20260929 20:19:41.908421 10324 mvcc.cc:205] Tried to move back new op lower bound from 7334761193021587456 to 7334761192861581312. Current Snapshot: MvccSnapshot[applied={T|T < 7334761193021587456}]
I20260929 20:19:41.909353 10322 mvcc.cc:205] Tried to move back new op lower bound from 7334761193021587456 to 7334761192861581312. Current Snapshot: MvccSnapshot[applied={T|T < 7334761193021587456}]
I20260929 20:19:41.920925 9834 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:19:41.921228 9834 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:19:41.924501 9834 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618: Sending DeleteTablet for 3 replicas of tablet 1f696f1d1200466ab391debd7d67b20d
I20260929 20:19:41.925904 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 9895
I20260929 20:19:41.925895 9958 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f696f1d1200466ab391debd7d67b20d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:19:41 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:34172
I20260929 20:19:41.925930 10225 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f696f1d1200466ab391debd7d67b20d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:19:41 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:55966
I20260929 20:19:41.926592 10092 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f696f1d1200466ab391debd7d67b20d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:19:41 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:36772
I20260929 20:19:41.938362 10333 tablet_replica.cc:330] T 1f696f1d1200466ab391debd7d67b20d P 9c0fe02fa1b14d13aa929b74e09d1f7f: stopping tablet replica
I20260929 20:19:41.938961 10335 tablet_replica.cc:330] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929: stopping tablet replica
I20260929 20:19:41.939515 10335 raft_consensus.cc:2229] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:19:41.940117 10333 raft_consensus.cc:2229] T 1f696f1d1200466ab391debd7d67b20d P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:19:41.940726 10333 raft_consensus.cc:2258] T 1f696f1d1200466ab391debd7d67b20d P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:19:41.941380 10332 tablet_replica.cc:330] T 1f696f1d1200466ab391debd7d67b20d P 6b2231cba4a6403d98b27afdb04948af: stopping tablet replica
I20260929 20:19:41.941509 10335 raft_consensus.cc:2258] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:19:41.941764 10332 raft_consensus.cc:2229] T 1f696f1d1200466ab391debd7d67b20d P 6b2231cba4a6403d98b27afdb04948af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:19:41.942241 10332 raft_consensus.cc:2258] T 1f696f1d1200466ab391debd7d67b20d P 6b2231cba4a6403d98b27afdb04948af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:19:41.943753 10335 ts_tablet_manager.cc:1905] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:19:41.943791 10333 ts_tablet_manager.cc:1905] T 1f696f1d1200466ab391debd7d67b20d P 9c0fe02fa1b14d13aa929b74e09d1f7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:19:41.947257 10332 ts_tablet_manager.cc:1905] T 1f696f1d1200466ab391debd7d67b20d P 6b2231cba4a6403d98b27afdb04948af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:19:41.951296 10335 ts_tablet_manager.cc:1918] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:19:41.951488 10335 log.cc:1196] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/wal/wals/1f696f1d1200466ab391debd7d67b20d
I20260929 20:19:41.952046 10335 ts_tablet_manager.cc:1939] T 1f696f1d1200466ab391debd7d67b20d P 693d5229f3094dd9896297d9ace72929: Deleting consensus metadata
I20260929 20:19:41.952176 10333 ts_tablet_manager.cc:1918] T 1f696f1d1200466ab391debd7d67b20d P 9c0fe02fa1b14d13aa929b74e09d1f7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:19:41.952419 10333 log.cc:1196] T 1f696f1d1200466ab391debd7d67b20d P 9c0fe02fa1b14d13aa929b74e09d1f7f: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/wal/wals/1f696f1d1200466ab391debd7d67b20d
I20260929 20:19:41.952998 10333 ts_tablet_manager.cc:1939] T 1f696f1d1200466ab391debd7d67b20d P 9c0fe02fa1b14d13aa929b74e09d1f7f: Deleting consensus metadata
I20260929 20:19:41.953795 9820 catalog_manager.cc:4841] TS 693d5229f3094dd9896297d9ace72929 (127.27.185.66:45885): tablet 1f696f1d1200466ab391debd7d67b20d (table test-table [id=c214059c648a43ea9ae9adcc08fc0af0]) successfully deleted
I20260929 20:19:41.954039 10332 ts_tablet_manager.cc:1918] T 1f696f1d1200466ab391debd7d67b20d P 6b2231cba4a6403d98b27afdb04948af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:19:41.954198 10332 log.cc:1196] T 1f696f1d1200466ab391debd7d67b20d P 6b2231cba4a6403d98b27afdb04948af: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/wal/wals/1f696f1d1200466ab391debd7d67b20d
I20260929 20:19:41.954696 10332 ts_tablet_manager.cc:1939] T 1f696f1d1200466ab391debd7d67b20d P 6b2231cba4a6403d98b27afdb04948af: Deleting consensus metadata
I20260929 20:19:41.955021 9818 catalog_manager.cc:4841] TS 9c0fe02fa1b14d13aa929b74e09d1f7f (127.27.185.65:33189): tablet 1f696f1d1200466ab391debd7d67b20d (table test-table [id=c214059c648a43ea9ae9adcc08fc0af0]) successfully deleted
I20260929 20:19:41.956344 9818 catalog_manager.cc:4841] TS 6b2231cba4a6403d98b27afdb04948af (127.27.185.67:42645): tablet 1f696f1d1200466ab391debd7d67b20d (table test-table [id=c214059c648a43ea9ae9adcc08fc0af0]) successfully deleted
Sep 29 20:19:41 dist-test-slave-4cx3 krb5kdc[9789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.65@KRBTEST.COM
I20260929 20:19:41.966813 10305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:41.927285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34186 (local address 127.27.185.65:33189)
0929 20:19:41.927530 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:41.927539 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:41.938177 (+ 10638us) server_negotiation.cc:407] Connection header received
0929 20:19:41.938668 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:41.938676 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:41.938772 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:41.938855 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:41.942576 (+ 3721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.943721 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.954331 (+ 10610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:41.955072 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:41.955182 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:41.960168 (+ 4986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:41.960185 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:41.960205 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:41.960250 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:41.963960 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:41.964659 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:41.964669 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:41.964673 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:41.964835 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:41.965195 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:41.965203 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:41.965206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:41.965645 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:41.965971 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:41.966154 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 20:19:41.966393 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 20:19:41.967698 10018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:19:42.058696 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 9895
I20260929 20:19:42.077209 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 10029
Sep 29 20:19:42 dist-test-slave-4cx3 krb5kdc[9789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.66@KRBTEST.COM
I20260929 20:19:42.094408 10302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:42.078035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36784 (local address 127.27.185.66:45885)
0929 20:19:42.078239 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:42.078245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:42.078267 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:19:42.078428 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:42.078434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:42.078549 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:42.078671 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:42.079952 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:42.081192 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:42.082443 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:42.083122 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:42.083254 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:42.088225 (+ 4971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:42.088242 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:42.088258 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:42.088300 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:42.091353 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:42.092415 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:42.092424 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:42.092432 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:42.092600 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:42.093007 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:42.093010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:42.093012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:42.093354 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:42.093596 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:42.093920 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 20:19:42.094086 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 20:19:42.095081 10152 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:19:42.206642 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 10029
I20260929 20:19:42.225636 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 10162
Sep 29 20:19:42 dist-test-slave-4cx3 krb5kdc[9789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.67@KRBTEST.COM
I20260929 20:19:42.240818 10307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:42.226401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55968 (local address 127.27.185.67:42645)
0929 20:19:42.226620 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:42.226625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:42.226647 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:19:42.226770 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:42.226774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:42.226835 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:42.226938 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:42.228299 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:42.229033 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:42.230304 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:42.230699 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:42.230837 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:42.235205 (+ 4368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:42.235220 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:42.235236 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:42.235276 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:42.237973 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:42.238660 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:42.238667 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:42.238669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:42.238812 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:42.239184 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:42.239191 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:42.239194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:42.239608 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:42.239903 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:42.240245 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 20:19:42.240491 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 20:19:42.241478 10285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:19:42.331260 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 10162
I20260929 20:19:42.348052 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 9802
I20260929 20:19:42.349247 9864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:19:42.457887 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 9802
I20260929 20:19:42.479192 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:33551
--webserver_interface=127.27.185.126
--webserver_port=46171
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:39637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:33551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:19:42.830938 10340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:19:42.831307 10340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:19:42.831501 10340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:19:42.838985 10340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:19:42.839130 10340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:19:42.839242 10340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:19:42.839344 10340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:19:42.839442 10340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:19:42.849713 10340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:39637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:33551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:33551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=46171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:19:42.851083 10340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:19:42.852761 10340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:19:42.876171 10346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:42.876225 10347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:42.876199 10349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:19:42.877136 10340 server_base.cc:1034] running on GCE node
I20260929 20:19:42.878103 10340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:19:42.879853 10340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:19:42.881102 10340 hybrid_clock.cc:648] HybridClock initialized: now 1790713182881046 us; error 61 us; skew 500 ppm
I20260929 20:19:42.881534 10340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:19:42 dist-test-slave-4cx3 krb5kdc[9789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713182, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:19:42.885191 10340 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:19:42.886950 10340 webserver.cc:460] Webserver started at http://127.27.185.126:46171/ using document root <none> and password file <none>
I20260929 20:19:42.887815 10340 fs_manager.cc:359] Metadata directory not provided
I20260929 20:19:42.887990 10340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:19:42.894169 10340 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 20:19:42.897683 10356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:42.899148 10340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:19:42.899345 10340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "e1cb929ec49c473a8f68aedf49e45618"
format_stamp: "Formatted at 2026-09-29 20:19:39 on dist-test-slave-4cx3"
server_key: "ecf1ffcd6da3cef7286d6f7ebe647e06"
server_key_iv: "6691a4912ef4a787ca18e1155de1e399"
server_key_version: "encryptionkey@0"
I20260929 20:19:42.900591 10340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:19:42.917798 10340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:19:42.921821 10340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:19:42.922113 10340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:19:42.941219 10340 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:33551
I20260929 20:19:42.941241 10407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:33551 every 8 connection(s)
I20260929 20:19:42.942811 10340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:19:42.947042 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 10340
I20260929 20:19:42.948026 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:33189
--local_ip_for_outbound_sockets=127.27.185.65
--tserver_master_addrs=127.27.185.126:33551
--webserver_port=45195
--webserver_interface=127.27.185.65
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:39637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:19:42.951607 10408 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:19:42.956903 10408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618: Bootstrap starting.
I20260929 20:19:42.977133 10408 log.cc:824] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618: Log is configured to *not* fsync() on all Append() calls
I20260929 20:19:42.989305 10408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:19:42.989868 10408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618: Bootstrap complete.
I20260929 20:19:42.999466 10408 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cb929ec49c473a8f68aedf49e45618" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33551 } }
I20260929 20:19:43.000324 10408 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1cb929ec49c473a8f68aedf49e45618, State: Initialized, Role: FOLLOWER
I20260929 20:19:43.000988 10408 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cb929ec49c473a8f68aedf49e45618" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33551 } }
I20260929 20:19:43.001235 10408 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:19:43.001362 10408 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:19:43.001554 10408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [term 1 FOLLOWER]: Advancing to term 2
I20260929 20:19:43.005614 10408 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cb929ec49c473a8f68aedf49e45618" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33551 } }
I20260929 20:19:43.006206 10408 leader_election.cc:302] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1cb929ec49c473a8f68aedf49e45618; no voters:
I20260929 20:19:43.006803 10408 leader_election.cc:288] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 20:19:43.006997 10412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 20:19:43.008262 10412 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [term 2 LEADER]: Becoming Leader. State: Replica: e1cb929ec49c473a8f68aedf49e45618, State: Running, Role: LEADER
I20260929 20:19:43.008967 10408 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:19:43.008864 10412 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cb929ec49c473a8f68aedf49e45618" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33551 } }
I20260929 20:19:43.014732 10414 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1cb929ec49c473a8f68aedf49e45618. Latest consensus state: current_term: 2 leader_uuid: "e1cb929ec49c473a8f68aedf49e45618" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cb929ec49c473a8f68aedf49e45618" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33551 } } }
I20260929 20:19:43.015174 10414 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [sys.catalog]: This master's current role is: LEADER
I20260929 20:19:43.014739 10413 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e1cb929ec49c473a8f68aedf49e45618" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cb929ec49c473a8f68aedf49e45618" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33551 } } }
I20260929 20:19:43.015563 10413 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [sys.catalog]: This master's current role is: LEADER
I20260929 20:19:43.023940 10422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:19:43.033795 10422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:19:43.035215 10422 catalog_manager.cc:1259] Loaded cluster ID: 789f194adc19451ab36e64680873055b
I20260929 20:19:43.035318 10422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:19:43.038551 10422 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 20:19:43.056787 10422 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 20:19:43.080430 10422 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:19:43.082340 10422 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618: Loaded TSK: 0
I20260929 20:19:43.083992 10422 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 20:19:43.329782 10410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:19:43.330093 10410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:19:43.330250 10410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:19:43.338198 10410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:19:43.338326 10410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:19:43.338495 10410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:19:43.348479 10410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:39637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:33189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=45195
--tserver_master_addrs=127.27.185.126:33551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:19:43.349814 10410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:19:43.351140 10410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:19:43.366080 10436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:43.366634 10438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:43.366681 10435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:19:43.368109 10410 server_base.cc:1034] running on GCE node
I20260929 20:19:43.368783 10410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:19:43.369807 10410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:19:43.371037 10410 hybrid_clock.cc:648] HybridClock initialized: now 1790713183370964 us; error 62 us; skew 500 ppm
I20260929 20:19:43.371352 10410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:19:43 dist-test-slave-4cx3 krb5kdc[9789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713183, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:19:43.374722 10410 init.cc:377] Logged in from keytab as kudu/127.27.185.65@KRBTEST.COM (short username kudu)
I20260929 20:19:43.376297 10410 webserver.cc:460] Webserver started at http://127.27.185.65:45195/ using document root <none> and password file <none>
I20260929 20:19:43.377031 10410 fs_manager.cc:359] Metadata directory not provided
I20260929 20:19:43.377172 10410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:19:43.382783 10410 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:19:43.385704 10445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:43.387110 10410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:19:43.387279 10410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f"
format_stamp: "Formatted at 2026-09-29 20:19:40 on dist-test-slave-4cx3"
server_key: "98de184ff94f09ec409c8855a63cce06"
server_key_iv: "5a76d19d0858b5bf110e7afe8bd1aec4"
server_key_version: "encryptionkey@0"
I20260929 20:19:43.388302 10410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:19:43.405937 10410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:19:43.410173 10410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:19:43.410445 10410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:19:43.411480 10410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:19:43.412976 10410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:19:43.413093 10410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:43.413223 10410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:19:43.413309 10410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:43.446702 10410 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:33189
I20260929 20:19:43.446731 10557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:33189 every 8 connection(s)
I20260929 20:19:43.448231 10410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:19:43.457197 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 10410
I20260929 20:19:43.457871 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:45885
--local_ip_for_outbound_sockets=127.27.185.66
--tserver_master_addrs=127.27.185.126:33551
--webserver_port=45403
--webserver_interface=127.27.185.66
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:39637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:19:43 dist-test-slave-4cx3 krb5kdc[9789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713183, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:19:43.474246 10561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:43.451217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.65:53581)
0929 20:19:43.452049 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:43.452093 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:43.453802 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:43.455079 (+ 1277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:43.455086 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:43.455659 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:19:43.456187 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:43.456204 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:43.457873 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:43.457879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:43.460111 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:43.460129 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:43.461155 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:43.461164 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:43.461493 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:43.462219 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:19:43.462252 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:19:43.465203 (+ 2951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:19:43.469987 (+ 4784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:43.469995 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:43.470008 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:43.470438 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:43.470988 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:43.470994 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:43.470998 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:43.471240 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:43.472247 (+ 1007us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:19:43.472253 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:19:43.472571 (+ 318us) client_negotiation.cc:769] Sending connection context
0929 20:19:43.472801 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 20:19:43.473216 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":381,"threads_started":1}
I20260929 20:19:43.474857 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:43.451092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:53581 (local address 127.27.185.126:33551)
0929 20:19:43.451760 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:43.451774 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:43.452214 (+ 440us) server_negotiation.cc:407] Connection header received
0929 20:19:43.454292 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:43.454305 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:43.454643 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:43.454951 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:43.456421 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:43.457695 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:43.460255 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:43.460917 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:43.461048 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:43.465563 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:43.465576 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:43.465600 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:43.465654 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:43.469815 (+ 4161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:43.470597 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:43.470609 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:43.470620 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:43.470843 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:43.471360 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:43.471364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:43.471368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:43.472116 (+ 748us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:43.472314 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:43.473081 (+ 767us) server_negotiation.cc:299] Negotiation successful
0929 20:19:43.473580 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":351,"threads_started":1}
I20260929 20:19:43.477579 10558 heartbeater.cc:347] Connected to a master server at 127.27.185.126:33551
I20260929 20:19:43.477916 10558 heartbeater.cc:464] Registering TS with master...
I20260929 20:19:43.478929 10558 heartbeater.cc:511] Master 127.27.185.126:33551 requested a full tablet report, sending...
I20260929 20:19:43.481309 10373 ts_manager.cc:194] Registered new tserver with Master: 9c0fe02fa1b14d13aa929b74e09d1f7f (127.27.185.65:33189)
I20260929 20:19:43.482746 10373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.65@KRBTEST.COM'} at 127.27.185.65:53581
W20260929 20:19:43.792338 10562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:19:43.792644 10562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:19:43.792829 10562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:19:43.800473 10562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:19:43.800637 10562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:19:43.800817 10562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:19:43.811684 10562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:39637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:45885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=45403
--tserver_master_addrs=127.27.185.126:33551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:19:43.813270 10562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:19:43.814740 10562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:19:43.829912 10570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:43.829952 10568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:43.829973 10567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:19:43.830595 10562 server_base.cc:1034] running on GCE node
I20260929 20:19:43.831267 10562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:19:43.832324 10562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:19:43.833547 10562 hybrid_clock.cc:648] HybridClock initialized: now 1790713183833500 us; error 41 us; skew 500 ppm
I20260929 20:19:43.833858 10562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:19:43 dist-test-slave-4cx3 krb5kdc[9789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713183, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:19:43.837004 10562 init.cc:377] Logged in from keytab as kudu/127.27.185.66@KRBTEST.COM (short username kudu)
I20260929 20:19:43.838436 10562 webserver.cc:460] Webserver started at http://127.27.185.66:45403/ using document root <none> and password file <none>
I20260929 20:19:43.839155 10562 fs_manager.cc:359] Metadata directory not provided
I20260929 20:19:43.839310 10562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:19:43.844673 10562 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:19:43.847321 10577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:43.848681 10562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:19:43.848856 10562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "693d5229f3094dd9896297d9ace72929"
format_stamp: "Formatted at 2026-09-29 20:19:41 on dist-test-slave-4cx3"
server_key: "4066633c4519859c978d89e6e6143178"
server_key_iv: "ef5f32b5229503a03de2cda201f3e8a5"
server_key_version: "encryptionkey@0"
I20260929 20:19:43.849848 10562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:19:43.862174 10562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:19:43.866099 10562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:19:43.866377 10562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:19:43.867316 10562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:19:43.868637 10562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:19:43.868753 10562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:43.868877 10562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:19:43.868963 10562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:43.901961 10562 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:45885
I20260929 20:19:43.902056 10689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:45885 every 8 connection(s)
I20260929 20:19:43.903512 10562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:19:43.905588 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 10562
I20260929 20:19:43.906225 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:42645
--local_ip_for_outbound_sockets=127.27.185.67
--tserver_master_addrs=127.27.185.126:33551
--webserver_port=41159
--webserver_interface=127.27.185.67
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:39637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:19:43 dist-test-slave-4cx3 krb5kdc[9789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713183, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:19:43.926146 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:43.906287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:33889 (local address 127.27.185.126:33551)
0929 20:19:43.906484 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:43.906490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:43.907958 (+ 1468us) server_negotiation.cc:407] Connection header received
0929 20:19:43.909983 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:43.909992 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:43.910103 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:43.910220 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:43.911969 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:43.913010 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:43.915419 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:43.915871 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:43.915973 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:43.920219 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:43.920224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:43.920227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:43.920262 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:43.922766 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:43.923574 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:43.923578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:43.923580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:43.923732 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:43.924299 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:43.924302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:43.924304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:43.924581 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:43.924736 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:43.925723 (+ 987us) server_negotiation.cc:299] Negotiation successful
0929 20:19:43.925907 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 20:19:43.928042 10692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:43.906892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.66:33889)
0929 20:19:43.907762 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:43.907811 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:43.909703 (+ 1892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:43.910467 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:43.910481 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:43.911010 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:19:43.911783 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:43.911808 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:43.913174 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:43.913179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:43.915225 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:43.915243 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:43.916098 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:43.916108 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:43.916439 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:43.917215 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:19:43.917249 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:19:43.919978 (+ 2729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:19:43.922921 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:43.922930 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:43.922948 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:43.923431 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:43.923877 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:43.923885 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:43.923891 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:43.924175 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:43.924718 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:19:43.924730 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:19:43.925257 (+ 527us) client_negotiation.cc:769] Sending connection context
0929 20:19:43.925638 (+ 381us) client_negotiation.cc:240] Negotiation successful
0929 20:19:43.926321 (+ 683us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":363,"threads_started":1}
I20260929 20:19:43.929742 10690 heartbeater.cc:347] Connected to a master server at 127.27.185.126:33551
I20260929 20:19:43.930065 10690 heartbeater.cc:464] Registering TS with master...
I20260929 20:19:43.930784 10690 heartbeater.cc:511] Master 127.27.185.126:33551 requested a full tablet report, sending...
I20260929 20:19:43.932353 10373 ts_manager.cc:194] Registered new tserver with Master: 693d5229f3094dd9896297d9ace72929 (127.27.185.66:45885)
I20260929 20:19:43.933166 10373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.66@KRBTEST.COM'} at 127.27.185.66:33889
W20260929 20:19:44.247896 10693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:19:44.248201 10693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:19:44.248379 10693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:19:44.255988 10693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:19:44.256115 10693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:19:44.256285 10693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:19:44.266270 10693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:39637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:42645
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=41159
--tserver_master_addrs=127.27.185.126:33551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:19:44.267721 10693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:19:44.269055 10693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:19:44.282239 10699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:44.282267 10698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:44.283005 10701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:19:44.283252 10693 server_base.cc:1034] running on GCE node
I20260929 20:19:44.283905 10693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:19:44.284893 10693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:19:44.286170 10693 hybrid_clock.cc:648] HybridClock initialized: now 1790713184286098 us; error 72 us; skew 500 ppm
I20260929 20:19:44.286520 10693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:19:44 dist-test-slave-4cx3 krb5kdc[9789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713184, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:19:44.289637 10693 init.cc:377] Logged in from keytab as kudu/127.27.185.67@KRBTEST.COM (short username kudu)
I20260929 20:19:44.291049 10693 webserver.cc:460] Webserver started at http://127.27.185.67:41159/ using document root <none> and password file <none>
I20260929 20:19:44.291842 10693 fs_manager.cc:359] Metadata directory not provided
I20260929 20:19:44.291998 10693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:19:44.297425 10693 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260929 20:19:44.300767 10708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:44.302191 10693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:19:44.302361 10693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "6b2231cba4a6403d98b27afdb04948af"
format_stamp: "Formatted at 2026-09-29 20:19:41 on dist-test-slave-4cx3"
server_key: "582f80d8ec070c77a1e67cbda5ac14f8"
server_key_iv: "a84718b5c4e35b47529a28591586a599"
server_key_version: "encryptionkey@0"
I20260929 20:19:44.303335 10693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:19:44.319060 10693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:19:44.322919 10693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:19:44.323192 10693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:19:44.324206 10693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:19:44.325520 10693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:19:44.325664 10693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:44.325801 10693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:19:44.325886 10693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:44.359491 10693 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:42645
I20260929 20:19:44.359555 10820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:42645 every 8 connection(s)
I20260929 20:19:44.361338 10693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:19:44.371666 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 10693
Sep 29 20:19:44 dist-test-slave-4cx3 krb5kdc[9789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713184, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:19:44.383893 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:44.363903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:50651 (local address 127.27.185.126:33551)
0929 20:19:44.364119 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:44.364126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:44.365282 (+ 1156us) server_negotiation.cc:407] Connection header received
0929 20:19:44.367069 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:44.367074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:44.367156 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:44.367248 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:44.368878 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.369658 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.371199 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.371896 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.372055 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:44.376918 (+ 4863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:44.376924 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:44.376927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:44.376960 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:44.380305 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:44.381227 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:44.381231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:44.381233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:44.381356 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:44.381870 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:44.381878 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:44.381884 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:44.382273 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:44.382431 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:44.383172 (+ 741us) server_negotiation.cc:299] Negotiation successful
0929 20:19:44.383470 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:19:44.384768 10823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:44.364434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.67:50651)
0929 20:19:44.365123 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:44.365161 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:44.366792 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:44.367525 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:44.367536 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:44.368068 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:19:44.368660 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:44.368677 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.369805 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.369809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:44.371025 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:44.371050 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.372111 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.372117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:44.372371 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:44.373230 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:19:44.373273 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:19:44.376698 (+ 3425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:19:44.380452 (+ 3754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:44.380459 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:44.380474 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:44.381097 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:44.381504 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:44.381511 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:44.381518 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:44.381749 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:44.382417 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:19:44.382424 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:19:44.382774 (+ 350us) client_negotiation.cc:769] Sending connection context
0929 20:19:44.383007 (+ 233us) client_negotiation.cc:240] Negotiation successful
0929 20:19:44.383477 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":265,"threads_started":1}
I20260929 20:19:44.386389 10821 heartbeater.cc:347] Connected to a master server at 127.27.185.126:33551
I20260929 20:19:44.386704 10821 heartbeater.cc:464] Registering TS with master...
I20260929 20:19:44.387508 10821 heartbeater.cc:511] Master 127.27.185.126:33551 requested a full tablet report, sending...
I20260929 20:19:44.389055 10373 ts_manager.cc:194] Registered new tserver with Master: 6b2231cba4a6403d98b27afdb04948af (127.27.185.67:42645)
I20260929 20:19:44.390168 10824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:44.372687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51654 (local address 127.27.185.126:33551)
0929 20:19:44.375451 (+ 2764us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:44.375458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:44.375775 (+ 317us) server_negotiation.cc:407] Connection header received
0929 20:19:44.375961 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:44.375965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:44.376059 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:44.376163 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:44.377662 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.378587 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.380662 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.381321 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.381412 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:44.384507 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:44.384514 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:44.384518 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:44.384556 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:44.387552 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:44.388249 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:44.388254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:44.388258 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:44.388379 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:44.388775 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:44.388778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:44.388780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:44.389071 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:44.389321 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:44.389665 (+ 344us) server_negotiation.cc:299] Negotiation successful
0929 20:19:44.389840 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2604,"thread_start_us":2405,"threads_started":1}
I20260929 20:19:44.390254 10373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.67@KRBTEST.COM'} at 127.27.185.67:50651
I20260929 20:19:44.392988 28389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:19:44.399672 10824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:44.393879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51658 (local address 127.27.185.126:33551)
0929 20:19:44.394035 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:44.394040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:44.394507 (+ 467us) server_negotiation.cc:407] Connection header received
0929 20:19:44.394679 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:44.394683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:44.394810 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:44.394924 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:19:44.395306 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.396089 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.397745 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.398181 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.398272 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:44.398683 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:19:44.399147 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:19:44.399253 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:44.399337 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 20:19:44.399453 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 20:19:44.400568 10372 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:51658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:19:44.408437 10372 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:19:44.445123 10834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:44.421048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55316 (local address 127.27.185.66:45885)
0929 20:19:44.433447 (+ 12399us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:44.433461 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:44.434616 (+ 1155us) server_negotiation.cc:407] Connection header received
0929 20:19:44.434987 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:44.435001 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:44.435324 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:44.435553 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:19:44.436798 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.439488 (+ 2690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.442616 (+ 3128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.443900 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.444080 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:44.444302 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:44.444643 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 20:19:44.444820 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12164,"thread_start_us":12420,"threads_started":1}
I20260929 20:19:44.445360 10831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:44.420329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58866 (local address 127.27.185.65:33189)
0929 20:19:44.433431 (+ 13102us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:44.433447 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:44.435638 (+ 2191us) server_negotiation.cc:407] Connection header received
0929 20:19:44.435745 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:44.435753 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:44.435959 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:44.436150 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:19:44.436800 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.438609 (+ 1809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.442449 (+ 3840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.444009 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.444152 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:44.444348 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:44.444870 (+ 522us) server_negotiation.cc:299] Negotiation successful
0929 20:19:44.445056 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12814,"thread_start_us":13148,"threads_started":1}
I20260929 20:19:44.445168 10832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:44.420269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:45885 (local address 127.0.0.1:55316)
0929 20:19:44.434388 (+ 14119us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:44.434442 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:44.434682 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:44.435895 (+ 1213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:44.435903 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:44.435945 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:19:44.436545 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:44.436565 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.439690 (+ 3125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.439703 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:44.442468 (+ 2765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:44.442478 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.444134 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.444140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:44.444351 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:44.444411 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 20:19:44.444566 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 20:19:44.444762 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13859,"mutex_wait_us":83,"thread_start_us":13694,"threads_started":1}
I20260929 20:19:44.446116 10833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:44.420288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:33189 (local address 127.0.0.1:58866)
0929 20:19:44.435335 (+ 15047us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:44.435354 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:44.435506 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:44.436148 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:44.436153 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:44.436179 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:19:44.436642 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:44.436653 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.438762 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.438767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:44.442278 (+ 3511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:44.442293 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.444505 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.444514 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:44.444678 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:44.444712 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 20:19:44.445546 (+ 834us) client_negotiation.cc:240] Negotiation successful
0929 20:19:44.445676 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14925,"mutex_wait_us":536,"spinlock_wait_cycles":22272,"thread_start_us":14775,"threads_started":1}
I20260929 20:19:44.449949 10493 tablet_service.cc:1467] Processing CreateTablet for tablet f5d2eb3e1a1a457d9e89384011a9fd99 (DEFAULT_TABLE table=test-table [id=153e3cbe4dab49a5b29dea1be2d098c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:19:44.449813 10625 tablet_service.cc:1467] Processing CreateTablet for tablet f5d2eb3e1a1a457d9e89384011a9fd99 (DEFAULT_TABLE table=test-table [id=153e3cbe4dab49a5b29dea1be2d098c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:19:44.451378 10493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5d2eb3e1a1a457d9e89384011a9fd99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:19:44.451363 10625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5d2eb3e1a1a457d9e89384011a9fd99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:19:44.453527 10835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:44.435942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41056 (local address 127.27.185.67:42645)
0929 20:19:44.441031 (+ 5089us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:44.441045 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:44.441712 (+ 667us) server_negotiation.cc:407] Connection header received
0929 20:19:44.441940 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:44.441948 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:44.442218 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:44.442837 (+ 619us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:19:44.443798 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.445355 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.449530 (+ 4175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.450970 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.451080 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:44.451284 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:44.452912 (+ 1628us) server_negotiation.cc:299] Negotiation successful
0929 20:19:44.453097 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4844,"thread_start_us":4635,"threads_started":1}
I20260929 20:19:44.454195 10836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:44.436902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:42645 (local address 127.0.0.1:41056)
0929 20:19:44.441592 (+ 4690us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:44.441607 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:44.441760 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:44.443111 (+ 1351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:44.443116 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:44.443142 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:19:44.443537 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:44.443549 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.445932 (+ 2383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.445941 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:44.449334 (+ 3393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:44.449349 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.451498 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.451504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:44.451680 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:44.451712 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 20:19:44.453706 (+ 1994us) client_negotiation.cc:240] Negotiation successful
0929 20:19:44.453859 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4568,"thread_start_us":4439,"threads_started":1}
I20260929 20:19:44.458616 10756 tablet_service.cc:1467] Processing CreateTablet for tablet f5d2eb3e1a1a457d9e89384011a9fd99 (DEFAULT_TABLE table=test-table [id=153e3cbe4dab49a5b29dea1be2d098c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:19:44.460114 10756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5d2eb3e1a1a457d9e89384011a9fd99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:19:44.467221 10837 tablet_bootstrap.cc:492] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f: Bootstrap starting.
I20260929 20:19:44.468015 10838 tablet_bootstrap.cc:492] T f5d2eb3e1a1a457d9e89384011a9fd99 P 693d5229f3094dd9896297d9ace72929: Bootstrap starting.
I20260929 20:19:44.471225 10837 tablet_bootstrap.cc:654] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f: Neither blocks nor log segments found. Creating new log.
I20260929 20:19:44.471640 10838 tablet_bootstrap.cc:654] T f5d2eb3e1a1a457d9e89384011a9fd99 P 693d5229f3094dd9896297d9ace72929: Neither blocks nor log segments found. Creating new log.
I20260929 20:19:44.472389 10837 log.cc:824] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:19:44.472898 10838 log.cc:824] T f5d2eb3e1a1a457d9e89384011a9fd99 P 693d5229f3094dd9896297d9ace72929: Log is configured to *not* fsync() on all Append() calls
I20260929 20:19:44.473536 10839 tablet_bootstrap.cc:492] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af: Bootstrap starting.
I20260929 20:19:44.476967 10838 tablet_bootstrap.cc:492] T f5d2eb3e1a1a457d9e89384011a9fd99 P 693d5229f3094dd9896297d9ace72929: No bootstrap required, opened a new log
I20260929 20:19:44.477229 10838 ts_tablet_manager.cc:1397] T f5d2eb3e1a1a457d9e89384011a9fd99 P 693d5229f3094dd9896297d9ace72929: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260929 20:19:44.477388 10837 tablet_bootstrap.cc:492] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f: No bootstrap required, opened a new log
I20260929 20:19:44.477665 10837 ts_tablet_manager.cc:1397] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20260929 20:19:44.478328 10839 tablet_bootstrap.cc:654] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af: Neither blocks nor log segments found. Creating new log.
I20260929 20:19:44.479514 10839 log.cc:824] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af: Log is configured to *not* fsync() on all Append() calls
I20260929 20:19:44.482429 10839 tablet_bootstrap.cc:492] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af: No bootstrap required, opened a new log
I20260929 20:19:44.482678 10839 ts_tablet_manager.cc:1397] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 20:19:44.484771 10838 raft_consensus.cc:348] T f5d2eb3e1a1a457d9e89384011a9fd99 P 693d5229f3094dd9896297d9ace72929 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 33189 } } peers { permanent_uuid: "693d5229f3094dd9896297d9ace72929" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 45885 } } peers { permanent_uuid: "6b2231cba4a6403d98b27afdb04948af" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 42645 } }
I20260929 20:19:44.485219 10838 raft_consensus.cc:374] T f5d2eb3e1a1a457d9e89384011a9fd99 P 693d5229f3094dd9896297d9ace72929 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:19:44.485313 10838 raft_consensus.cc:729] T f5d2eb3e1a1a457d9e89384011a9fd99 P 693d5229f3094dd9896297d9ace72929 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 693d5229f3094dd9896297d9ace72929, State: Initialized, Role: FOLLOWER
I20260929 20:19:44.485920 10838 consensus_queue.cc:260] T f5d2eb3e1a1a457d9e89384011a9fd99 P 693d5229f3094dd9896297d9ace72929 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 33189 } } peers { permanent_uuid: "693d5229f3094dd9896297d9ace72929" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 45885 } } peers { permanent_uuid: "6b2231cba4a6403d98b27afdb04948af" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 42645 } }
I20260929 20:19:44.486588 10558 heartbeater.cc:503] Master 127.27.185.126:33551 was elected leader, sending a full tablet report...
I20260929 20:19:44.488023 10837 raft_consensus.cc:348] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 33189 } } peers { permanent_uuid: "693d5229f3094dd9896297d9ace72929" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 45885 } } peers { permanent_uuid: "6b2231cba4a6403d98b27afdb04948af" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 42645 } }
I20260929 20:19:44.488431 10837 raft_consensus.cc:374] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:19:44.488515 10837 raft_consensus.cc:729] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c0fe02fa1b14d13aa929b74e09d1f7f, State: Initialized, Role: FOLLOWER
I20260929 20:19:44.489058 10837 consensus_queue.cc:260] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 33189 } } peers { permanent_uuid: "693d5229f3094dd9896297d9ace72929" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 45885 } } peers { permanent_uuid: "6b2231cba4a6403d98b27afdb04948af" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 42645 } }
I20260929 20:19:44.493263 10690 heartbeater.cc:503] Master 127.27.185.126:33551 was elected leader, sending a full tablet report...
I20260929 20:19:44.494058 10838 ts_tablet_manager.cc:1428] T f5d2eb3e1a1a457d9e89384011a9fd99 P 693d5229f3094dd9896297d9ace72929: Time spent starting tablet: real 0.017s user 0.014s sys 0.004s
I20260929 20:19:44.494864 10839 raft_consensus.cc:348] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 33189 } } peers { permanent_uuid: "693d5229f3094dd9896297d9ace72929" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 45885 } } peers { permanent_uuid: "6b2231cba4a6403d98b27afdb04948af" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 42645 } }
I20260929 20:19:44.495306 10839 raft_consensus.cc:374] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:19:44.495424 10839 raft_consensus.cc:729] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b2231cba4a6403d98b27afdb04948af, State: Initialized, Role: FOLLOWER
I20260929 20:19:44.496042 10837 ts_tablet_manager.cc:1428] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f: Time spent starting tablet: real 0.018s user 0.015s sys 0.004s
I20260929 20:19:44.496127 10839 consensus_queue.cc:260] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 33189 } } peers { permanent_uuid: "693d5229f3094dd9896297d9ace72929" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 45885 } } peers { permanent_uuid: "6b2231cba4a6403d98b27afdb04948af" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 42645 } }
I20260929 20:19:44.497231 10821 heartbeater.cc:503] Master 127.27.185.126:33551 was elected leader, sending a full tablet report...
I20260929 20:19:44.499156 10839 ts_tablet_manager.cc:1428] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
W20260929 20:19:44.612910 10822 tablet.cc:2369] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:19:44.655527 10691 tablet.cc:2369] T f5d2eb3e1a1a457d9e89384011a9fd99 P 693d5229f3094dd9896297d9ace72929: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:19:44.696470 10845 raft_consensus.cc:482] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:19:44.696825 10845 raft_consensus.cc:504] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 33189 } } peers { permanent_uuid: "693d5229f3094dd9896297d9ace72929" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 45885 } } peers { permanent_uuid: "6b2231cba4a6403d98b27afdb04948af" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 42645 } }
I20260929 20:19:44.698632 10845 leader_election.cc:288] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c0fe02fa1b14d13aa929b74e09d1f7f (127.27.185.65:33189), 693d5229f3094dd9896297d9ace72929 (127.27.185.66:45885)
W20260929 20:19:44.700481 10559 tablet.cc:2369] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:19:44.706168 10831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:44.699099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:45989 (local address 127.27.185.65:33189)
0929 20:19:44.699465 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:44.699470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:44.699652 (+ 182us) server_negotiation.cc:407] Connection header received
0929 20:19:44.699890 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:44.699896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:44.699982 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:44.700075 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:19:44.700559 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.701563 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.704059 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.705314 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.705482 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:44.705560 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:44.705806 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 20:19:44.705905 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20260929 20:19:44.706240 10823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:44.699071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:33189 (local address 127.27.185.67:45989)
0929 20:19:44.699506 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:44.699546 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:44.699723 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:44.700125 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:44.700131 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:44.700190 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:19:44.700388 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:44.700402 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.701698 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.701702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:44.703910 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:44.703925 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.705457 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.705462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:44.705600 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:44.705656 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 20:19:44.705772 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 20:19:44.705918 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":68}
I20260929 20:19:44.706513 10846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:44.699124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:45885 (local address 127.27.185.67:35783)
0929 20:19:44.699954 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:44.699971 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:44.700108 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:44.700462 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:44.700468 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:44.700495 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:19:44.700886 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:44.700899 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.702091 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.702096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:44.704144 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:44.704154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.705756 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.705764 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:44.705925 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:44.705953 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 20:19:44.706036 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 20:19:44.706156 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":482,"threads_started":1}
I20260929 20:19:44.706887 10834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:44.699135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:35783 (local address 127.27.185.66:45885)
0929 20:19:44.699344 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:44.699350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:44.700084 (+ 734us) server_negotiation.cc:407] Connection header received
0929 20:19:44.700245 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:44.700251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:44.700325 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:44.700422 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:19:44.701073 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.701954 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.704276 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.705558 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.705686 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:44.705758 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:44.706076 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 20:19:44.706211 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:19:44.707356 10513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5d2eb3e1a1a457d9e89384011a9fd99" candidate_uuid: "6b2231cba4a6403d98b27afdb04948af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f" is_pre_election: true
I20260929 20:19:44.707765 10645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5d2eb3e1a1a457d9e89384011a9fd99" candidate_uuid: "6b2231cba4a6403d98b27afdb04948af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "693d5229f3094dd9896297d9ace72929" is_pre_election: true
I20260929 20:19:44.707995 10513 raft_consensus.cc:2454] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b2231cba4a6403d98b27afdb04948af in term 0.
I20260929 20:19:44.708138 10645 raft_consensus.cc:2454] T f5d2eb3e1a1a457d9e89384011a9fd99 P 693d5229f3094dd9896297d9ace72929 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b2231cba4a6403d98b27afdb04948af in term 0.
I20260929 20:19:44.708726 10709 leader_election.cc:302] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b2231cba4a6403d98b27afdb04948af, 9c0fe02fa1b14d13aa929b74e09d1f7f; no voters:
I20260929 20:19:44.709196 10845 raft_consensus.cc:2781] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:19:44.709352 10845 raft_consensus.cc:482] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:19:44.709467 10845 raft_consensus.cc:3037] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:19:44.711802 10845 raft_consensus.cc:504] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 33189 } } peers { permanent_uuid: "693d5229f3094dd9896297d9ace72929" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 45885 } } peers { permanent_uuid: "6b2231cba4a6403d98b27afdb04948af" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 42645 } }
I20260929 20:19:44.712370 10845 leader_election.cc:288] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af [CANDIDATE]: Term 1 election: Requested vote from peers 9c0fe02fa1b14d13aa929b74e09d1f7f (127.27.185.65:33189), 693d5229f3094dd9896297d9ace72929 (127.27.185.66:45885)
I20260929 20:19:44.712893 10513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5d2eb3e1a1a457d9e89384011a9fd99" candidate_uuid: "6b2231cba4a6403d98b27afdb04948af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f"
I20260929 20:19:44.712893 10645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5d2eb3e1a1a457d9e89384011a9fd99" candidate_uuid: "6b2231cba4a6403d98b27afdb04948af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "693d5229f3094dd9896297d9ace72929"
I20260929 20:19:44.713111 10513 raft_consensus.cc:3037] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:19:44.713156 10645 raft_consensus.cc:3037] T f5d2eb3e1a1a457d9e89384011a9fd99 P 693d5229f3094dd9896297d9ace72929 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:19:44.716449 10645 raft_consensus.cc:2454] T f5d2eb3e1a1a457d9e89384011a9fd99 P 693d5229f3094dd9896297d9ace72929 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b2231cba4a6403d98b27afdb04948af in term 1.
I20260929 20:19:44.716449 10513 raft_consensus.cc:2454] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b2231cba4a6403d98b27afdb04948af in term 1.
I20260929 20:19:44.716930 10711 leader_election.cc:302] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 693d5229f3094dd9896297d9ace72929, 6b2231cba4a6403d98b27afdb04948af, 9c0fe02fa1b14d13aa929b74e09d1f7f; no voters:
I20260929 20:19:44.717348 10845 raft_consensus.cc:2781] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:19:44.717777 10845 raft_consensus.cc:686] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af [term 1 LEADER]: Becoming Leader. State: Replica: 6b2231cba4a6403d98b27afdb04948af, State: Running, Role: LEADER
I20260929 20:19:44.718261 10845 consensus_queue.cc:237] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 33189 } } peers { permanent_uuid: "693d5229f3094dd9896297d9ace72929" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 45885 } } peers { permanent_uuid: "6b2231cba4a6403d98b27afdb04948af" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 42645 } }
I20260929 20:19:44.723047 10371 catalog_manager.cc:5495] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b2231cba4a6403d98b27afdb04948af (127.27.185.67). New cstate: current_term: 1 leader_uuid: "6b2231cba4a6403d98b27afdb04948af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 33189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "693d5229f3094dd9896297d9ace72929" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 45885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b2231cba4a6403d98b27afdb04948af" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 42645 } health_report { overall_health: HEALTHY } } }
I20260929 20:19:44.742908 10835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:44.737901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41068 (local address 127.27.185.67:42645)
0929 20:19:44.738087 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:44.738091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:44.738161 (+ 70us) server_negotiation.cc:407] Connection header received
0929 20:19:44.738339 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:44.738343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:44.738419 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:44.738553 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:19:44.738869 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.739651 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.740943 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.741337 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.741429 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:44.741857 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:19:44.742382 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:19:44.742502 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:44.742640 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 20:19:44.742708 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:19:44.751015 10513 raft_consensus.cc:1261] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 1 FOLLOWER]: Refusing update from remote peer 6b2231cba4a6403d98b27afdb04948af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:19:44.751005 10645 raft_consensus.cc:1261] T f5d2eb3e1a1a457d9e89384011a9fd99 P 693d5229f3094dd9896297d9ace72929 [term 1 FOLLOWER]: Refusing update from remote peer 6b2231cba4a6403d98b27afdb04948af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:19:44.752038 10847 consensus_queue.cc:1035] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 33189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:19:44.752381 10845 consensus_queue.cc:1035] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af [LEADER]: Connected to new peer: Peer: permanent_uuid: "693d5229f3094dd9896297d9ace72929" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 45885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:19:44.764341 10852 mvcc.cc:205] Tried to move back new op lower bound from 7334761204730294272 to 7334761204609556480. Current Snapshot: MvccSnapshot[applied={T|T < 7334761204730294272}]
I20260929 20:19:44.765995 10850 mvcc.cc:205] Tried to move back new op lower bound from 7334761204730294272 to 7334761204609556480. Current Snapshot: MvccSnapshot[applied={T|T < 7334761204730294272}]
I20260929 20:19:44.778206 10373 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:51658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:19:44.778472 10373 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:51658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:19:44.781952 10373 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618: Sending DeleteTablet for 3 replicas of tablet f5d2eb3e1a1a457d9e89384011a9fd99
I20260929 20:19:44.783236 10625 tablet_service.cc:1514] Processing DeleteTablet for tablet f5d2eb3e1a1a457d9e89384011a9fd99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:19:44 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:55316
I20260929 20:19:44.783236 10493 tablet_service.cc:1514] Processing DeleteTablet for tablet f5d2eb3e1a1a457d9e89384011a9fd99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:19:44 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:58866
I20260929 20:19:44.783329 10756 tablet_service.cc:1514] Processing DeleteTablet for tablet f5d2eb3e1a1a457d9e89384011a9fd99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:19:44 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:41056
I20260929 20:19:44.785014 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 10410
I20260929 20:19:44.808665 10831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:44.786435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58880 (local address 127.27.185.65:33189)
0929 20:19:44.786680 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:44.786689 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:44.786711 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:19:44.786784 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:44.786791 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:44.786891 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:44.786997 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:44.789185 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.791006 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.793592 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.794519 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.794641 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:44.799129 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:44.799155 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:44.799182 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:44.799256 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:44.804488 (+ 5232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:44.805664 (+ 1176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:44.805679 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:44.805685 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:44.805927 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:44.806412 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:44.806421 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:44.806427 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:44.807093 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:44.807730 (+ 637us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:44.807935 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 20:19:44.808165 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 20:19:44.810154 10553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:19:44.810391 10861 tablet_replica.cc:330] T f5d2eb3e1a1a457d9e89384011a9fd99 P 693d5229f3094dd9896297d9ace72929: stopping tablet replica
I20260929 20:19:44.811062 10861 raft_consensus.cc:2229] T f5d2eb3e1a1a457d9e89384011a9fd99 P 693d5229f3094dd9896297d9ace72929 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:19:44.811931 10861 raft_consensus.cc:2258] T f5d2eb3e1a1a457d9e89384011a9fd99 P 693d5229f3094dd9896297d9ace72929 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:19:44.813187 10862 tablet_replica.cc:330] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af: stopping tablet replica
I20260929 20:19:44.815210 10861 ts_tablet_manager.cc:1905] T f5d2eb3e1a1a457d9e89384011a9fd99 P 693d5229f3094dd9896297d9ace72929: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:19:44.820533 10862 raft_consensus.cc:2229] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:19:44.821451 10862 raft_consensus.cc:2258] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:19:44.823833 10862 ts_tablet_manager.cc:1905] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:19:44.826756 10861 ts_tablet_manager.cc:1918] T f5d2eb3e1a1a457d9e89384011a9fd99 P 693d5229f3094dd9896297d9ace72929: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:19:44.826939 10861 log.cc:1196] T f5d2eb3e1a1a457d9e89384011a9fd99 P 693d5229f3094dd9896297d9ace72929: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/wal/wals/f5d2eb3e1a1a457d9e89384011a9fd99
I20260929 20:19:44.827571 10861 ts_tablet_manager.cc:1939] T f5d2eb3e1a1a457d9e89384011a9fd99 P 693d5229f3094dd9896297d9ace72929: Deleting consensus metadata
I20260929 20:19:44.829674 10359 catalog_manager.cc:4841] TS 693d5229f3094dd9896297d9ace72929 (127.27.185.66:45885): tablet f5d2eb3e1a1a457d9e89384011a9fd99 (table test-table [id=153e3cbe4dab49a5b29dea1be2d098c1]) successfully deleted
I20260929 20:19:44.833400 10862 ts_tablet_manager.cc:1918] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:19:44.833599 10862 log.cc:1196] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/wal/wals/f5d2eb3e1a1a457d9e89384011a9fd99
I20260929 20:19:44.834226 10862 ts_tablet_manager.cc:1939] T f5d2eb3e1a1a457d9e89384011a9fd99 P 6b2231cba4a6403d98b27afdb04948af: Deleting consensus metadata
I20260929 20:19:44.836396 10357 catalog_manager.cc:4841] TS 6b2231cba4a6403d98b27afdb04948af (127.27.185.67:42645): tablet f5d2eb3e1a1a457d9e89384011a9fd99 (table test-table [id=153e3cbe4dab49a5b29dea1be2d098c1]) successfully deleted
I20260929 20:19:44.914499 10860 tablet_replica.cc:330] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f: stopping tablet replica
I20260929 20:19:44.914790 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 10410
I20260929 20:19:44.929858 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 10562
W20260929 20:19:44.930591 10357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.185.65:33189 (error 108)
I20260929 20:19:44.935866 10836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:44.935243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:33189 (local address 127.0.0.1:58888)
0929 20:19:44.935544 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:44.935678 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:33189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 20:19:44.936349 10357 catalog_manager.cc:4584] TS 9c0fe02fa1b14d13aa929b74e09d1f7f (127.27.185.65:33189): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f5d2eb3e1a1a457d9e89384011a9fd99: Network error: Client connection negotiation failed: client connection to 127.27.185.65:33189: connect: Connection refused (error 111)
I20260929 20:19:44.946761 10834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:44.930714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55320 (local address 127.27.185.66:45885)
0929 20:19:44.931454 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:44.931464 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:44.931489 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:19:44.931568 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:44.931577 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:44.931680 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:44.931797 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:44.934124 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.934874 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.936431 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:44.936908 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:44.937044 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:44.939772 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:44.939783 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:44.939797 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:44.939828 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:44.943454 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:44.944369 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:44.944379 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:44.944383 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:44.944520 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:44.944930 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:44.944935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:44.944939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:44.945395 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:44.945679 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:44.945985 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 20:19:44.946203 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523}
I20260929 20:19:44.947554 10685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:19:44.990097 10836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:44.989530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:33189 (local address 127.0.0.1:58890)
0929 20:19:44.989786 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:44.989922 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:33189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 20:19:44.991271 10836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:44.990836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:33189 (local address 127.0.0.1:58902)
0929 20:19:44.991006 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:44.991099 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:33189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 20:19:45.044328 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 10562
I20260929 20:19:45.052177 10836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:45.051601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:33189 (local address 127.0.0.1:58916)
0929 20:19:45.051841 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:45.051965 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:33189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 20:19:45.053485 10836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:45.053026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:33189 (local address 127.0.0.1:58918)
0929 20:19:45.053215 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:45.053318 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:33189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 20:19:45.063021 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 10693
I20260929 20:19:45.080068 10835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:45.063824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41084 (local address 127.27.185.67:42645)
0929 20:19:45.064002 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:45.064010 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:45.064249 (+ 239us) server_negotiation.cc:407] Connection header received
0929 20:19:45.064453 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:45.064457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:45.064527 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:45.064618 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:45.066421 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:45.067233 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:45.069254 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:45.069723 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:45.069829 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:45.073595 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:45.073608 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:45.073624 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:45.073665 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:45.076891 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:45.077844 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:45.077851 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:45.077857 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:45.078028 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:45.078489 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:45.078492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:45.078494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:45.078855 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:45.079083 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:45.079518 (+ 435us) server_negotiation.cc:299] Negotiation successful
0929 20:19:45.079696 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 20:19:45.080736 10816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:19:45.134258 10836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:45.133657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:33189 (local address 127.0.0.1:58928)
0929 20:19:45.133924 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:45.134041 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:33189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 20:19:45.135377 10836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:45.134934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:33189 (local address 127.0.0.1:58944)
0929 20:19:45.135165 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:45.135250 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:33189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260929 20:19:45.203128 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 10693
I20260929 20:19:45.223114 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 10340
I20260929 20:19:45.224280 10403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:19:45.338644 10836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:45.337977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:33189 (local address 127.0.0.1:58948)
0929 20:19:45.338315 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:45.338442 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:33189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260929 20:19:45.338624 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 10340
I20260929 20:19:45.361351 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:33551
--webserver_interface=127.27.185.126
--webserver_port=46171
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:39637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:33551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:19:45.700711 10868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:19:45.701079 10868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:19:45.701234 10868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:19:45.708570 10868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:19:45.708727 10868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:19:45.708837 10868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:19:45.708941 10868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:19:45.709024 10868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:19:45.719169 10868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:39637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:33551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:33551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=46171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:19:45.720621 10868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:19:45.722011 10868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:19:45.734025 10873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:45.734084 10874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:45.734086 10876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:19:45.734997 10868 server_base.cc:1034] running on GCE node
I20260929 20:19:45.735847 10868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:19:45.737308 10868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:19:45.738550 10868 hybrid_clock.cc:648] HybridClock initialized: now 1790713185738483 us; error 53 us; skew 500 ppm
I20260929 20:19:45.738907 10868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:19:45 dist-test-slave-4cx3 krb5kdc[9789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713185, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:19:45.742144 10868 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:19:45.743638 10868 webserver.cc:460] Webserver started at http://127.27.185.126:46171/ using document root <none> and password file <none>
I20260929 20:19:45.744320 10868 fs_manager.cc:359] Metadata directory not provided
I20260929 20:19:45.744426 10868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:19:45.750136 10868 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 20:19:45.753312 10883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:45.754781 10868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:19:45.754968 10868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "e1cb929ec49c473a8f68aedf49e45618"
format_stamp: "Formatted at 2026-09-29 20:19:39 on dist-test-slave-4cx3"
server_key: "ecf1ffcd6da3cef7286d6f7ebe647e06"
server_key_iv: "6691a4912ef4a787ca18e1155de1e399"
server_key_version: "encryptionkey@0"
I20260929 20:19:45.756088 10868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:19:45.769552 10868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:19:45.773561 10868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:19:45.773844 10868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:19:45.793210 10868 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:33551
I20260929 20:19:45.793220 10934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:33551 every 8 connection(s)
I20260929 20:19:45.794893 10868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:19:45.802264 10935 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:19:45.804180 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 10868
I20260929 20:19:45.805202 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:33189
--local_ip_for_outbound_sockets=127.27.185.65
--tserver_master_addrs=127.27.185.126:33551
--webserver_port=45195
--webserver_interface=127.27.185.65
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:39637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:19:45.806807 10935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618: Bootstrap starting.
I20260929 20:19:45.829883 10935 log.cc:824] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618: Log is configured to *not* fsync() on all Append() calls
I20260929 20:19:45.846122 10935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:19:45.846693 10935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618: Bootstrap complete.
I20260929 20:19:45.855937 10935 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cb929ec49c473a8f68aedf49e45618" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33551 } }
I20260929 20:19:45.856882 10935 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1cb929ec49c473a8f68aedf49e45618, State: Initialized, Role: FOLLOWER
I20260929 20:19:45.857498 10935 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cb929ec49c473a8f68aedf49e45618" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33551 } }
I20260929 20:19:45.857741 10935 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:19:45.857867 10935 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:19:45.858048 10935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [term 2 FOLLOWER]: Advancing to term 3
I20260929 20:19:45.862391 10935 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cb929ec49c473a8f68aedf49e45618" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33551 } }
I20260929 20:19:45.862907 10935 leader_election.cc:302] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1cb929ec49c473a8f68aedf49e45618; no voters:
I20260929 20:19:45.863639 10935 leader_election.cc:288] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 20:19:45.863778 10939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 20:19:45.864852 10939 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [term 3 LEADER]: Becoming Leader. State: Replica: e1cb929ec49c473a8f68aedf49e45618, State: Running, Role: LEADER
I20260929 20:19:45.865494 10939 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cb929ec49c473a8f68aedf49e45618" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33551 } }
I20260929 20:19:45.865931 10935 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:19:45.868013 10941 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1cb929ec49c473a8f68aedf49e45618. Latest consensus state: current_term: 3 leader_uuid: "e1cb929ec49c473a8f68aedf49e45618" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cb929ec49c473a8f68aedf49e45618" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33551 } } }
I20260929 20:19:45.868415 10941 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [sys.catalog]: This master's current role is: LEADER
I20260929 20:19:45.868610 10940 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e1cb929ec49c473a8f68aedf49e45618" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cb929ec49c473a8f68aedf49e45618" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33551 } } }
I20260929 20:19:45.868970 10940 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1cb929ec49c473a8f68aedf49e45618 [sys.catalog]: This master's current role is: LEADER
I20260929 20:19:45.880450 10947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:19:45.889366 10947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:19:45.890914 10947 catalog_manager.cc:1259] Loaded cluster ID: 789f194adc19451ab36e64680873055b
I20260929 20:19:45.891103 10947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 20:19:45.893234 10947 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe25fd619ad google::LogMessage::Fail() at ??:0
@ 0x7fe25fd6488a google::LogMessage::SendToLog() at ??:0
@ 0x7fe25fd614de google::LogMessage::Flush() at ??:0
@ 0x7fe25fd62a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe275798512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fe275795976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe27581d610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fe27581d439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe2629ad345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe2629b091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fe2629b0799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe26298beaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe265c656db start_thread at ??:0
@ 0x7fe25ebfb61f clone at ??:0
W20260929 20:19:46.216074 10937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:19:46.216378 10937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:19:46.216480 10937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:19:46.223794 10937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:19:46.223941 10937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:19:46.224148 10937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:19:46.233997 10937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:39637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:33189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=45195
--tserver_master_addrs=127.27.185.126:33551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:19:46.235252 10937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:19:46.236636 10937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:19:46.249953 10977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:46.249976 10976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:46.250068 10979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:19:46.250838 10937 server_base.cc:1034] running on GCE node
I20260929 20:19:46.251543 10937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:19:46.252604 10937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:19:46.253839 10937 hybrid_clock.cc:648] HybridClock initialized: now 1790713186253790 us; error 43 us; skew 500 ppm
I20260929 20:19:46.254204 10937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:19:46 dist-test-slave-4cx3 krb5kdc[9789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713186, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:19:46.257383 10937 init.cc:377] Logged in from keytab as kudu/127.27.185.65@KRBTEST.COM (short username kudu)
I20260929 20:19:46.258857 10937 webserver.cc:460] Webserver started at http://127.27.185.65:45195/ using document root <none> and password file <none>
I20260929 20:19:46.259594 10937 fs_manager.cc:359] Metadata directory not provided
I20260929 20:19:46.259706 10937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:19:46.265038 10937 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:19:46.269876 10986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:46.271355 10937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:19:46.271566 10937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f"
format_stamp: "Formatted at 2026-09-29 20:19:40 on dist-test-slave-4cx3"
server_key: "98de184ff94f09ec409c8855a63cce06"
server_key_iv: "5a76d19d0858b5bf110e7afe8bd1aec4"
server_key_version: "encryptionkey@0"
I20260929 20:19:46.272557 10937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:19:46.290630 10937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:19:46.294780 10937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:19:46.295017 10937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:19:46.296005 10937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:19:46.297943 10993 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260929 20:19:46.304225 10937 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 20:19:46.304351 10937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260929 20:19:46.304467 10937 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 20:19:46.308470 10937 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 20:19:46.308566 10937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.003s sys 0.000s
I20260929 20:19:46.308966 10993 tablet_bootstrap.cc:492] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f: Bootstrap starting.
I20260929 20:19:46.337112 10993 log.cc:824] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:19:46.350811 10937 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:33189
I20260929 20:19:46.350884 10993 tablet_bootstrap.cc:492] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:19:46.350850 11100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:33189 every 8 connection(s)
I20260929 20:19:46.351616 10993 tablet_bootstrap.cc:492] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f: Bootstrap complete.
I20260929 20:19:46.352284 10937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:19:46.353137 10993 ts_tablet_manager.cc:1397] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f: Time spent bootstrapping tablet: real 0.045s user 0.034s sys 0.008s
I20260929 20:19:46.353252 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 10937
I20260929 20:19:46.354005 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:45885
--local_ip_for_outbound_sockets=127.27.185.66
--tserver_master_addrs=127.27.185.126:33551
--webserver_port=45403
--webserver_interface=127.27.185.66
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:39637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:19:46.360138 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:46.357045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.65:37425)
0929 20:19:46.358213 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:46.358636 (+ 423us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":471,"threads_started":1}
W20260929 20:19:46.361009 11101 heartbeater.cc:644] Failed to heartbeat to 127.27.185.126:33551 (0 consecutive failures): Network error: Failed to ping master at 127.27.185.126:33551: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
I20260929 20:19:46.361927 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:46.361562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.65:44087)
0929 20:19:46.361727 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:46.361813 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 20:19:46.362802 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:46.362440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.65:34783)
0929 20:19:46.362585 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:46.362676 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 20:19:46.363098 11101 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 20:19:46.366542 10993 raft_consensus.cc:348] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 33189 } } peers { permanent_uuid: "693d5229f3094dd9896297d9ace72929" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 45885 } } peers { permanent_uuid: "6b2231cba4a6403d98b27afdb04948af" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 42645 } }
I20260929 20:19:46.367519 10993 raft_consensus.cc:729] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c0fe02fa1b14d13aa929b74e09d1f7f, State: Initialized, Role: FOLLOWER
I20260929 20:19:46.368196 10993 consensus_queue.cc:260] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 33189 } } peers { permanent_uuid: "693d5229f3094dd9896297d9ace72929" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 45885 } } peers { permanent_uuid: "6b2231cba4a6403d98b27afdb04948af" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 42645 } }
I20260929 20:19:46.369853 10993 ts_tablet_manager.cc:1428] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f: Time spent starting tablet: real 0.017s user 0.008s sys 0.007s
I20260929 20:19:46.370042 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:46.369590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.65:47113)
0929 20:19:46.369780 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:46.369887 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 20:19:46.371104 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:46.370663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.65:44635)
0929 20:19:46.370841 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:46.370939 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 20:19:46.699513 11104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:19:46.699816 11104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:19:46.700032 11104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:19:46.708158 11104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:19:46.708290 11104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:19:46.708694 11104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:19:46.719540 11104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:39637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:45885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=45403
--tserver_master_addrs=127.27.185.126:33551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:19:46.721413 11104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:19:46.722898 11104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:19:46.737620 11110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:46.737689 11111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:46.738238 11113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:19:46.738612 11104 server_base.cc:1034] running on GCE node
I20260929 20:19:46.739444 11104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:19:46.740537 11104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:19:46.741799 11104 hybrid_clock.cc:648] HybridClock initialized: now 1790713186741750 us; error 60 us; skew 500 ppm
I20260929 20:19:46.742166 11104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:19:46 dist-test-slave-4cx3 krb5kdc[9789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713186, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:19:46.745997 11104 init.cc:377] Logged in from keytab as kudu/127.27.185.66@KRBTEST.COM (short username kudu)
I20260929 20:19:46.747614 11104 webserver.cc:460] Webserver started at http://127.27.185.66:45403/ using document root <none> and password file <none>
I20260929 20:19:46.748322 11104 fs_manager.cc:359] Metadata directory not provided
I20260929 20:19:46.748487 11104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:19:46.754526 11104 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 20:19:46.757711 11120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:46.759261 11104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:19:46.759564 11104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "693d5229f3094dd9896297d9ace72929"
format_stamp: "Formatted at 2026-09-29 20:19:41 on dist-test-slave-4cx3"
server_key: "4066633c4519859c978d89e6e6143178"
server_key_iv: "ef5f32b5229503a03de2cda201f3e8a5"
server_key_version: "encryptionkey@0"
I20260929 20:19:46.760674 11104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:19:46.789778 11104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:19:46.795068 11104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:19:46.795431 11104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:19:46.796921 11104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:19:46.798463 11104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:19:46.798599 11104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:46.798741 11104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:19:46.798832 11104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:46.834187 11104 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:45885
I20260929 20:19:46.834240 11232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:45885 every 8 connection(s)
I20260929 20:19:46.835963 11104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:19:46.841194 11235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:46.839128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.66:36793)
0929 20:19:46.839953 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:46.840231 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":352,"threads_started":1}
W20260929 20:19:46.842059 11233 heartbeater.cc:644] Failed to heartbeat to 127.27.185.126:33551 (0 consecutive failures): Network error: Failed to ping master at 127.27.185.126:33551: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
I20260929 20:19:46.842937 11235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:46.842562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.66:50627)
0929 20:19:46.842728 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:46.842808 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 20:19:46.843895 11235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:46.843564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.66:36543)
0929 20:19:46.843704 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:46.843785 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 20:19:46.844213 11233 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 20:19:46.845347 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 11104
I20260929 20:19:46.845921 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:42645
--local_ip_for_outbound_sockets=127.27.185.67
--tserver_master_addrs=127.27.185.126:33551
--webserver_port=41159
--webserver_interface=127.27.185.67
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:39637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:19:47.179674 11236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:19:47.180008 11236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:19:47.180171 11236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:19:47.187361 11236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:19:47.187562 11236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:19:47.187731 11236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:19:47.197387 11236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:39637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:42645
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=41159
--tserver_master_addrs=127.27.185.126:33551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:19:47.198706 11236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:19:47.200156 11236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:19:47.213124 11241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:47.213205 11244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:19:47.213165 11242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:19:47.214293 11236 server_base.cc:1034] running on GCE node
I20260929 20:19:47.214890 11236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:19:47.215884 11236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:19:47.217085 11236 hybrid_clock.cc:648] HybridClock initialized: now 1790713187217047 us; error 40 us; skew 500 ppm
I20260929 20:19:47.217355 11236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:19:47 dist-test-slave-4cx3 krb5kdc[9789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713187, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:19:47.220602 11236 init.cc:377] Logged in from keytab as kudu/127.27.185.67@KRBTEST.COM (short username kudu)
I20260929 20:19:47.222079 11236 webserver.cc:460] Webserver started at http://127.27.185.67:41159/ using document root <none> and password file <none>
I20260929 20:19:47.222783 11236 fs_manager.cc:359] Metadata directory not provided
I20260929 20:19:47.222939 11236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:19:47.228400 11236 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:19:47.231072 11251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:47.232506 11236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:19:47.232697 11236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "6b2231cba4a6403d98b27afdb04948af"
format_stamp: "Formatted at 2026-09-29 20:19:41 on dist-test-slave-4cx3"
server_key: "582f80d8ec070c77a1e67cbda5ac14f8"
server_key_iv: "a84718b5c4e35b47529a28591586a599"
server_key_version: "encryptionkey@0"
I20260929 20:19:47.233688 11236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:19:47.248370 11236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:19:47.252801 11236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:19:47.253098 11236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:19:47.254071 11236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:19:47.255460 11236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:19:47.255589 11236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:47.255718 11236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:19:47.255816 11236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:19:47.288916 11236 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:42645
I20260929 20:19:47.288965 11363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:42645 every 8 connection(s)
I20260929 20:19:47.290428 11236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:19:47.295235 11366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:47.293439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.67:56545)
0929 20:19:47.294187 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:47.294399 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":283,"threads_started":1}
W20260929 20:19:47.296005 11364 heartbeater.cc:644] Failed to heartbeat to 127.27.185.126:33551 (0 consecutive failures): Network error: Failed to ping master at 127.27.185.126:33551: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
I20260929 20:19:47.296871 11366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:47.296552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.67:45041)
0929 20:19:47.296698 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:47.296773 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 20:19:47.297658 11366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:47.297377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.67:46487)
0929 20:19:47.297490 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:47.297563 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 20:19:47.297920 11364 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 20:19:47.299069 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 11236
I20260929 20:19:47.372872 11368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:47.372044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.65:52803)
0929 20:19:47.372580 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:47.372668 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":248,"threads_started":1}
I20260929 20:19:47.845533 11369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:47.844746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.66:41137)
0929 20:19:47.845193 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:47.845312 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":208,"threads_started":1}
I20260929 20:19:48.062979 11370 raft_consensus.cc:482] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:19:48.063235 11370 raft_consensus.cc:504] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 33189 } } peers { permanent_uuid: "693d5229f3094dd9896297d9ace72929" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 45885 } } peers { permanent_uuid: "6b2231cba4a6403d98b27afdb04948af" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 42645 } }
I20260929 20:19:48.065068 11370 leader_election.cc:288] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 693d5229f3094dd9896297d9ace72929 (127.27.185.66:45885), 6b2231cba4a6403d98b27afdb04948af (127.27.185.67:42645)
Sep 29 20:19:48 dist-test-slave-4cx3 krb5kdc[9789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713186, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for kudu/127.27.185.66@KRBTEST.COM
Sep 29 20:19:48 dist-test-slave-4cx3 krb5kdc[9789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713186, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for kudu/127.27.185.67@KRBTEST.COM
I20260929 20:19:48.105734 11372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:48.065557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:40025 (local address 127.27.185.66:45885)
0929 20:19:48.077912 (+ 12355us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:48.077927 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:48.077975 (+ 48us) server_negotiation.cc:407] Connection header received
0929 20:19:48.080352 (+ 2377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:48.080362 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:48.080809 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:48.081250 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:48.084384 (+ 3134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:48.085811 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:48.087721 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:48.088458 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:48.088666 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:48.096508 (+ 7842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:48.096517 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:48.096535 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:48.096578 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:48.100670 (+ 4092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:48.101866 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:48.101874 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:48.101882 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:48.102125 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:48.102806 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:48.102812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:48.102816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:48.103555 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:48.103749 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:48.104880 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0929 20:19:48.105314 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12148,"thread_start_us":11982,"threads_started":1}
I20260929 20:19:48.105734 11374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:48.065747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:56795 (local address 127.27.185.67:42645)
0929 20:19:48.076488 (+ 10741us) server_negotiation.cc:206] Beginning negotiation
0929 20:19:48.076515 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 20:19:48.077658 (+ 1143us) server_negotiation.cc:407] Connection header received
0929 20:19:48.080201 (+ 2543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:19:48.080214 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:19:48.080808 (+ 594us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:19:48.081229 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:19:48.084383 (+ 3154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:48.085806 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:48.087721 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:19:48.088360 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:19:48.088507 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:48.097015 (+ 8508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:19:48.097026 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:19:48.097046 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:19:48.097097 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:19:48.100719 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:48.101865 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:48.101874 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:48.101882 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:48.102125 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:19:48.102806 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:19:48.102812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:19:48.102816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:19:48.103576 (+ 760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:19:48.103753 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 20:19:48.104880 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0929 20:19:48.105313 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10493,"thread_start_us":10366,"threads_started":1}
I20260929 20:19:48.105729 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:48.065455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:45885 (local address 127.27.185.65:40025)
0929 20:19:48.076898 (+ 11443us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:48.076953 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:48.077719 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:48.081454 (+ 3735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:48.081467 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:48.082793 (+ 1326us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:19:48.084126 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:48.084157 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:48.086032 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:48.086037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:48.087574 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:48.087587 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:48.088667 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:48.088674 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:48.088871 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:48.093380 (+ 4509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:19:48.093409 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:19:48.096145 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:19:48.100859 (+ 4714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:48.100869 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:48.100887 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:48.101704 (+ 817us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:48.102249 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:48.102255 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:48.102258 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:48.102646 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:48.103735 (+ 1089us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:19:48.103744 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:19:48.104245 (+ 501us) client_negotiation.cc:769] Sending connection context
0929 20:19:48.104611 (+ 366us) client_negotiation.cc:240] Negotiation successful
0929 20:19:48.105306 (+ 695us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11279,"mutex_wait_us":416,"spinlock_wait_cycles":70528,"thread_start_us":11565,"threads_started":1}
I20260929 20:19:48.105778 11371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:48.065522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:42645 (local address 127.27.185.65:56795)
0929 20:19:48.077328 (+ 11806us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:48.077357 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 20:19:48.077650 (+ 293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:19:48.081454 (+ 3804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:19:48.081467 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:19:48.082793 (+ 1326us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:19:48.084126 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:48.084157 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:48.086021 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:48.086025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:48.087574 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:19:48.087588 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:19:48.088567 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:19:48.088574 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:19:48.088897 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:19:48.093376 (+ 4479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:19:48.093406 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:19:48.096704 (+ 3298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:19:48.100859 (+ 4155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:48.100869 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:48.100887 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:48.101704 (+ 817us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:48.102249 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:19:48.102254 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:19:48.102258 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:19:48.102677 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:19:48.103735 (+ 1058us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:19:48.103744 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:19:48.104245 (+ 501us) client_negotiation.cc:769] Sending connection context
0929 20:19:48.104611 (+ 366us) client_negotiation.cc:240] Negotiation successful
0929 20:19:48.105177 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11685,"mutex_wait_us":393,"spinlock_wait_cycles":298624,"thread_start_us":11050,"threads_started":1}
W20260929 20:19:48.109383 10989 leader_election.cc:341] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 693d5229f3094dd9896297d9ace72929 (127.27.185.66:45885): Not found: Tablet not found: f5d2eb3e1a1a457d9e89384011a9fd99
W20260929 20:19:48.109735 10987 leader_election.cc:341] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6b2231cba4a6403d98b27afdb04948af (127.27.185.67:42645): Not found: Tablet not found: f5d2eb3e1a1a457d9e89384011a9fd99
I20260929 20:19:48.109915 10987 leader_election.cc:302] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c0fe02fa1b14d13aa929b74e09d1f7f; no voters: 693d5229f3094dd9896297d9ace72929, 6b2231cba4a6403d98b27afdb04948af
I20260929 20:19:48.110312 11370 raft_consensus.cc:2726] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 20:19:48.299242 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:48.298391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.67:51755)
0929 20:19:48.298844 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:48.298988 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":217,"threads_started":1}
I20260929 20:19:48.374511 11371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:48.374034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.65:42851)
0929 20:19:48.374242 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:48.374347 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 20:19:48.847501 11376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:48.846565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.66:40213)
0929 20:19:48.847151 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:48.847269 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":343,"threads_started":1}
I20260929 20:19:49.300879 11377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:49.300145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.67:39971)
0929 20:19:49.300634 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:49.300715 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":284,"threads_started":1}
I20260929 20:19:49.376494 11378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:49.375624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.65:40079)
0929 20:19:49.376236 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:49.376318 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":285,"threads_started":1}
I20260929 20:19:49.849018 11379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:49.848347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.66:57147)
0929 20:19:49.848772 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:49.848852 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":230,"threads_started":1}
I20260929 20:19:50.026240 11380 raft_consensus.cc:482] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:19:50.026453 11380 raft_consensus.cc:504] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 33189 } } peers { permanent_uuid: "693d5229f3094dd9896297d9ace72929" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 45885 } } peers { permanent_uuid: "6b2231cba4a6403d98b27afdb04948af" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 42645 } }
I20260929 20:19:50.027163 11380 leader_election.cc:288] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 693d5229f3094dd9896297d9ace72929 (127.27.185.66:45885), 6b2231cba4a6403d98b27afdb04948af (127.27.185.67:42645)
W20260929 20:19:50.028136 10987 leader_election.cc:341] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6b2231cba4a6403d98b27afdb04948af (127.27.185.67:42645): Not found: Tablet not found: f5d2eb3e1a1a457d9e89384011a9fd99
W20260929 20:19:50.028295 10989 leader_election.cc:341] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 693d5229f3094dd9896297d9ace72929 (127.27.185.66:45885): Not found: Tablet not found: f5d2eb3e1a1a457d9e89384011a9fd99
I20260929 20:19:50.028425 10989 leader_election.cc:302] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c0fe02fa1b14d13aa929b74e09d1f7f; no voters: 693d5229f3094dd9896297d9ace72929, 6b2231cba4a6403d98b27afdb04948af
I20260929 20:19:50.028650 11380 raft_consensus.cc:2726] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 20:19:50.302438 11381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:50.301731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.67:56307)
0929 20:19:50.302161 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:50.302240 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":222,"threads_started":1}
I20260929 20:19:50.378432 11382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:50.377463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.65:55151)
0929 20:19:50.377982 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:50.378175 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":234,"threads_started":1}
I20260929 20:19:50.850608 11383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:50.849898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.66:57461)
0929 20:19:50.850340 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:50.850419 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":222,"threads_started":1}
I20260929 20:19:51.304370 11384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:51.303442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.67:33267)
0929 20:19:51.304121 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:51.304198 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":441,"threads_started":1}
I20260929 20:19:51.380270 11385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:51.379580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.65:54683)
0929 20:19:51.380010 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:51.380100 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":228,"threads_started":1}
I20260929 20:19:51.852329 11386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:51.851648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.66:46397)
0929 20:19:51.852081 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:51.852161 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":205,"threads_started":1}
I20260929 20:19:52.306007 11387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:52.305328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.67:60629)
0929 20:19:52.305762 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:52.305852 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":229,"threads_started":1}
I20260929 20:19:52.381847 11388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:52.381130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.65:47815)
0929 20:19:52.381597 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:52.381682 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":265,"threads_started":1}
I20260929 20:19:52.413964 11389 raft_consensus.cc:482] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:19:52.414206 11389 raft_consensus.cc:504] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 33189 } } peers { permanent_uuid: "693d5229f3094dd9896297d9ace72929" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 45885 } } peers { permanent_uuid: "6b2231cba4a6403d98b27afdb04948af" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 42645 } }
I20260929 20:19:52.414856 11389 leader_election.cc:288] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 693d5229f3094dd9896297d9ace72929 (127.27.185.66:45885), 6b2231cba4a6403d98b27afdb04948af (127.27.185.67:42645)
W20260929 20:19:52.415796 10987 leader_election.cc:341] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6b2231cba4a6403d98b27afdb04948af (127.27.185.67:42645): Not found: Tablet not found: f5d2eb3e1a1a457d9e89384011a9fd99
W20260929 20:19:52.415939 10989 leader_election.cc:341] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 693d5229f3094dd9896297d9ace72929 (127.27.185.66:45885): Not found: Tablet not found: f5d2eb3e1a1a457d9e89384011a9fd99
I20260929 20:19:52.416108 10989 leader_election.cc:302] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c0fe02fa1b14d13aa929b74e09d1f7f; no voters: 693d5229f3094dd9896297d9ace72929, 6b2231cba4a6403d98b27afdb04948af
I20260929 20:19:52.416349 11389 raft_consensus.cc:2726] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 20:19:52.853896 11390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:52.853188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.66:48891)
0929 20:19:52.853618 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:52.853705 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":224,"threads_started":1}
I20260929 20:19:53.307943 11391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:53.307018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.67:48845)
0929 20:19:53.307603 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:53.307693 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":318,"threads_started":1}
I20260929 20:19:53.383772 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:53.382905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.65:55707)
0929 20:19:53.383351 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:53.383572 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":231,"threads_started":1}
I20260929 20:19:53.855602 11393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:53.854766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.66:47917)
0929 20:19:53.855284 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:53.855374 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":234,"threads_started":1}
I20260929 20:19:54.309662 11394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:54.308855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.67:33215)
0929 20:19:54.309381 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:54.309469 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":228,"threads_started":1}
I20260929 20:19:54.385458 11395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:54.384729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.65:46699)
0929 20:19:54.385206 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:54.385292 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":243,"threads_started":1}
I20260929 20:19:54.658058 11396 raft_consensus.cc:482] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:19:54.658262 11396 raft_consensus.cc:504] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 33189 } } peers { permanent_uuid: "693d5229f3094dd9896297d9ace72929" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 45885 } } peers { permanent_uuid: "6b2231cba4a6403d98b27afdb04948af" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 42645 } }
I20260929 20:19:54.658900 11396 leader_election.cc:288] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 693d5229f3094dd9896297d9ace72929 (127.27.185.66:45885), 6b2231cba4a6403d98b27afdb04948af (127.27.185.67:42645)
W20260929 20:19:54.659879 10987 leader_election.cc:341] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6b2231cba4a6403d98b27afdb04948af (127.27.185.67:42645): Not found: Tablet not found: f5d2eb3e1a1a457d9e89384011a9fd99
W20260929 20:19:54.660079 10989 leader_election.cc:341] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 693d5229f3094dd9896297d9ace72929 (127.27.185.66:45885): Not found: Tablet not found: f5d2eb3e1a1a457d9e89384011a9fd99
I20260929 20:19:54.660248 10989 leader_election.cc:302] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c0fe02fa1b14d13aa929b74e09d1f7f; no voters: 693d5229f3094dd9896297d9ace72929, 6b2231cba4a6403d98b27afdb04948af
I20260929 20:19:54.660516 11396 raft_consensus.cc:2726] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 20:19:54.857331 11397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:54.856491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.66:55847)
0929 20:19:54.856970 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:54.857076 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":225,"threads_started":1}
I20260929 20:19:55.311322 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:55.310555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.67:58775)
0929 20:19:55.311062 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:55.311144 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":304,"threads_started":1}
I20260929 20:19:55.387133 11399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:55.386396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.65:52165)
0929 20:19:55.386870 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:55.386954 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":265,"threads_started":1}
I20260929 20:19:55.859107 11400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:55.858400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.66:36047)
0929 20:19:55.858868 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:55.858954 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":233,"threads_started":1}
I20260929 20:19:56.312840 11401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:56.312178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.67:37095)
0929 20:19:56.312613 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:56.312690 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":208,"threads_started":1}
I20260929 20:19:56.388854 11402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:56.388176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.65:50453)
0929 20:19:56.388611 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:56.388690 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":233,"threads_started":1}
I20260929 20:19:56.501072 11403 raft_consensus.cc:482] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:19:56.501252 11403 raft_consensus.cc:504] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c0fe02fa1b14d13aa929b74e09d1f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 33189 } } peers { permanent_uuid: "693d5229f3094dd9896297d9ace72929" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 45885 } } peers { permanent_uuid: "6b2231cba4a6403d98b27afdb04948af" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 42645 } }
I20260929 20:19:56.501890 11403 leader_election.cc:288] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 693d5229f3094dd9896297d9ace72929 (127.27.185.66:45885), 6b2231cba4a6403d98b27afdb04948af (127.27.185.67:42645)
W20260929 20:19:56.502866 10989 leader_election.cc:341] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 693d5229f3094dd9896297d9ace72929 (127.27.185.66:45885): Not found: Tablet not found: f5d2eb3e1a1a457d9e89384011a9fd99
W20260929 20:19:56.503060 10987 leader_election.cc:341] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6b2231cba4a6403d98b27afdb04948af (127.27.185.67:42645): Not found: Tablet not found: f5d2eb3e1a1a457d9e89384011a9fd99
I20260929 20:19:56.503172 10987 leader_election.cc:302] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c0fe02fa1b14d13aa929b74e09d1f7f; no voters: 693d5229f3094dd9896297d9ace72929, 6b2231cba4a6403d98b27afdb04948af
I20260929 20:19:56.503413 11403 raft_consensus.cc:2726] T f5d2eb3e1a1a457d9e89384011a9fd99 P 9c0fe02fa1b14d13aa929b74e09d1f7f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 20:19:56.860766 11404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:56.860087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.66:40485)
0929 20:19:56.860503 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:56.860593 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":216,"threads_started":1}
I20260929 20:19:57.314417 11405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:57.313735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.67:39853)
0929 20:19:57.314162 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:57.314245 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":220,"threads_started":1}
I20260929 20:19:57.390573 11406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:57.389810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.65:56635)
0929 20:19:57.390269 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:57.390367 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":249,"threads_started":1}
I20260929 20:19:57.862633 11407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:57.861689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.66:40175)
0929 20:19:57.862302 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:57.862400 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":310,"threads_started":1}
I20260929 20:19:58.316315 11408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:58.315603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.67:42449)
0929 20:19:58.316070 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:58.316155 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":222,"threads_started":1}
I20260929 20:19:58.392354 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:58.391634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.65:54125)
0929 20:19:58.392094 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:58.392176 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":238,"threads_started":1}
I20260929 20:19:58.864588 11410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:58.863711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.66:53393)
0929 20:19:58.864285 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:58.864405 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":285,"threads_started":1}
I20260929 20:19:59.317881 11411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:59.317205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.67:55337)
0929 20:19:59.317629 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:59.317709 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":233,"threads_started":1}
I20260929 20:19:59.394019 11412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:59.393319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.65:44717)
0929 20:19:59.393746 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:59.393824 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":226,"threads_started":1}
I20260929 20:19:59.866263 11413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:19:59.865547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.66:34287)
0929 20:19:59.866001 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 20:19:59.866099 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":232,"threads_started":1}
I20260929 20:20:00.319519 11414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:00.318749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.67:33301)
0929 20:20:00.319180 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:00.319278 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":229,"threads_started":1}
I20260929 20:20:00.395802 11415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:00.394991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.65:49195)
0929 20:20:00.395538 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:00.395624 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":239,"threads_started":1}
I20260929 20:20:00.867933 11416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:00.867130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.66:45999)
0929 20:20:00.867626 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:00.867715 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":280,"threads_started":1}
I20260929 20:20:01.321249 11417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:01.320498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.67:34645)
0929 20:20:01.320961 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:01.321054 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":238,"threads_started":1}
I20260929 20:20:01.397853 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:01.397107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.65:33543)
0929 20:20:01.397593 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:01.397683 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":258,"threads_started":1}
I20260929 20:20:01.869869 11419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:01.869051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.66:37007)
0929 20:20:01.869557 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:01.869672 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":248,"threads_started":1}
I20260929 20:20:02.311275 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 10937
I20260929 20:20:02.322943 11421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:02.322275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33551 (local address 127.27.185.67:48515)
0929 20:20:02.322702 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:02.322786 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":234,"threads_started":1}
I20260929 20:20:02.332296 11420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:02.312710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37788 (local address 127.27.185.65:33189)
0929 20:20:02.313228 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:02.313236 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:02.313259 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:20:02.313318 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:02.313325 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:02.313535 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:02.313707 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:20:02.315069 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:02.316096 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:02.317325 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:02.317875 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:02.318026 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:02.326522 (+ 8496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:20:02.326534 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:20:02.326549 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:20:02.326589 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:20:02.329466 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:02.330158 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:02.330166 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:02.330168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:02.330301 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:02.330679 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:02.330686 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:02.330689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:02.331054 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:20:02.331370 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:02.331701 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 20:20:02.331986 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":222,"threads_started":1}
I20260929 20:20:02.333544 11096 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:20:02.424285 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 10937
I20260929 20:20:02.442615 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 11104
I20260929 20:20:02.456283 11424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:02.443443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38170 (local address 127.27.185.66:45885)
0929 20:20:02.444036 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:02.444044 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:02.444063 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:20:02.444128 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:02.444137 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:02.444196 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:02.444308 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:20:02.445686 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:02.446496 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:02.447818 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:02.448259 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:02.448398 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:02.451102 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:20:02.451108 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:20:02.451111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:20:02.451143 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:20:02.453608 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:02.454293 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:02.454299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:02.454303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:02.454448 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:02.454813 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:02.454820 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:02.454823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:02.455093 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:20:02.455365 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:02.455674 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 20:20:02.455888 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":307,"threads_started":1}
I20260929 20:20:02.456974 11228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:20:02.545945 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 11104
I20260929 20:20:02.563949 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 11236
I20260929 20:20:02.578490 11426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:02.564803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38500 (local address 127.27.185.67:42645)
0929 20:20:02.565419 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:02.565424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:02.565440 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:20:02.565494 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:02.565498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:02.565564 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:02.565685 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:20:02.567356 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:02.568253 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:02.569556 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:02.570021 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:02.570163 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:02.572886 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:20:02.572892 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:20:02.572897 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:20:02.572930 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:20:02.575732 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:02.576458 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:02.576465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:02.576468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:02.576614 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:02.577003 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:02.577010 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:02.577013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:02.577291 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:20:02.577611 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:02.577878 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 20:20:02.578098 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":342,"threads_started":1}
I20260929 20:20:02.579279 11359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:20:02.662546 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 11236
2026-09-29T20:20:02Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24411 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:20:02 dist-test-slave-4cx3 krb5kdc[11430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:20:02 dist-test-slave-4cx3 krb5kdc[11430](info): set up 2 sockets
Sep 29 20:20:02 dist-test-slave-4cx3 krb5kdc[11430](info): commencing operation
krb5kdc: starting...
W20260929 20:20:03.798933 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:20:03 dist-test-slave-4cx3 krb5kdc[11430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:20:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:20:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
I20260929 20:20:04.048511 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:41609
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:43943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:41609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:20:04.380985 11443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:20:04.381283 11443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:20:04.381446 11443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:20:04.388940 11443 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 20:20:04.389051 11443 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 20:20:04.389119 11443 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 20:20:04.389194 11443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:20:04.389279 11443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:20:04.389420 11443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:20:04.389503 11443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:20:04.389580 11443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:20:04.399694 11443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:43943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:41609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:41609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:20:04.401011 11443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:20:04.402369 11443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:20:04.414659 11449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:20:04.414634 11448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:20:04.414595 11451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:20:04.415268 11443 server_base.cc:1034] running on GCE node
I20260929 20:20:04.416115 11443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:20:04.417508 11443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:20:04.418752 11443 hybrid_clock.cc:648] HybridClock initialized: now 1790713204418699 us; error 52 us; skew 500 ppm
I20260929 20:20:04.419097 11443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:20:04 dist-test-slave-4cx3 krb5kdc[11430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713204, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:20:04.422255 11443 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:20:04.423759 11443 webserver.cc:460] Webserver started at http://127.27.185.126:43017/ using document root <none> and password file <none>
I20260929 20:20:04.424453 11443 fs_manager.cc:359] Metadata directory not provided
I20260929 20:20:04.424594 11443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:20:04.424916 11443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:20:04.427976 11443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "5562e2eff8df42f69e44202013a0c134"
format_stamp: "Formatted at 2026-09-29 20:20:04 on dist-test-slave-4cx3"
server_key: "581fafcbb6b9fb55652cf1fa48dbd301"
server_key_iv: "ea41d01177b62999a128ee64f88d6725"
server_key_version: "encryptionkey@0"
I20260929 20:20:04.428759 11443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "5562e2eff8df42f69e44202013a0c134"
format_stamp: "Formatted at 2026-09-29 20:20:04 on dist-test-slave-4cx3"
server_key: "581fafcbb6b9fb55652cf1fa48dbd301"
server_key_iv: "ea41d01177b62999a128ee64f88d6725"
server_key_version: "encryptionkey@0"
I20260929 20:20:04.433930 11443 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 20:20:04.437322 11458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:20:04.438581 11443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:20:04.438761 11443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "5562e2eff8df42f69e44202013a0c134"
format_stamp: "Formatted at 2026-09-29 20:20:04 on dist-test-slave-4cx3"
server_key: "581fafcbb6b9fb55652cf1fa48dbd301"
server_key_iv: "ea41d01177b62999a128ee64f88d6725"
server_key_version: "encryptionkey@0"
I20260929 20:20:04.438977 11443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:20:04.466794 11443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:20:04.470920 11443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:20:04.471256 11443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:20:04.490345 11443 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:41609
I20260929 20:20:04.490368 11509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:41609 every 8 connection(s)
I20260929 20:20:04.491815 11443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:20:04.495828 11510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:20:04.502143 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 11443
I20260929 20:20:04.502372 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:20:04.503232 28389 external_mini_cluster.cc:1428] Setting key 723585e19c93d17f4f06dbd062f1f92b
I20260929 20:20:04.507233 11510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134: Bootstrap starting.
I20260929 20:20:04.511773 11510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134: Neither blocks nor log segments found. Creating new log.
I20260929 20:20:04.513182 11510 log.cc:824] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134: Log is configured to *not* fsync() on all Append() calls
I20260929 20:20:04.517011 11510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134: No bootstrap required, opened a new log
Sep 29 20:20:04 dist-test-slave-4cx3 krb5kdc[11430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:20:04.527508 11510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5562e2eff8df42f69e44202013a0c134" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41609 } }
I20260929 20:20:04.528043 11510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:20:04.528152 11510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5562e2eff8df42f69e44202013a0c134, State: Initialized, Role: FOLLOWER
I20260929 20:20:04.528870 11510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5562e2eff8df42f69e44202013a0c134" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41609 } }
I20260929 20:20:04.529181 11510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:20:04.529383 11510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:20:04.529582 11510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:20:04.531116 11513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:04.505397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33684 (local address 127.27.185.126:41609)
0929 20:20:04.506193 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:04.506206 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:04.506250 (+ 44us) server_negotiation.cc:407] Connection header received
0929 20:20:04.508080 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:04.508104 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:04.508593 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:04.509089 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:20:04.511831 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:04.513931 (+ 2100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:04.516014 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:04.517061 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:04.517950 (+ 889us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:04.521999 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:20:04.522015 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:20:04.522037 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:20:04.522087 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:20:04.526197 (+ 4110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:04.526928 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:04.526936 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:04.526945 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:04.527159 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:04.527604 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:04.527608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:04.527611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:04.528392 (+ 781us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:20:04.528730 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:04.529131 (+ 401us) server_negotiation.cc:299] Negotiation successful
0929 20:20:04.529639 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":363,"threads_started":1}
I20260929 20:20:04.532954 11510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5562e2eff8df42f69e44202013a0c134" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41609 } }
I20260929 20:20:04.533471 11510 leader_election.cc:302] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5562e2eff8df42f69e44202013a0c134; no voters:
I20260929 20:20:04.534119 11510 leader_election.cc:288] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:20:04.534263 11515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:20:04.535085 11515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [term 1 LEADER]: Becoming Leader. State: Replica: 5562e2eff8df42f69e44202013a0c134, State: Running, Role: LEADER
I20260929 20:20:04.535804 11515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5562e2eff8df42f69e44202013a0c134" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41609 } }
I20260929 20:20:04.536507 11510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:20:04.542948 11517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5562e2eff8df42f69e44202013a0c134. Latest consensus state: current_term: 1 leader_uuid: "5562e2eff8df42f69e44202013a0c134" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5562e2eff8df42f69e44202013a0c134" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41609 } } }
I20260929 20:20:04.543366 11517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [sys.catalog]: This master's current role is: LEADER
I20260929 20:20:04.543260 11516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5562e2eff8df42f69e44202013a0c134" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5562e2eff8df42f69e44202013a0c134" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41609 } } }
I20260929 20:20:04.543704 11516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [sys.catalog]: This master's current role is: LEADER
W20260929 20:20:04.549758 11529 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:20:04.549902 11529 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:20:04.550065 11530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:20:04.552341 11530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:20:04.561190 11530 catalog_manager.cc:1347] Generated new cluster ID: 890e623071744b2ea5e2b009ddb3566f
I20260929 20:20:04.561312 11530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:20:04.574573 11530 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:20:04.576092 11530 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:20:04.586416 11530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134: Generated new TSK 0
I20260929 20:20:04.587340 11530 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
I20260929 20:20:04.668167 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:0
--local_ip_for_outbound_sockets=127.27.185.65
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:41609
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:43943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:20:05.026571 11536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:20:05.026857 11536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:20:05.026950 11536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:20:05.034422 11536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:20:05.034538 11536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:20:05.034652 11536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:20:05.045260 11536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:43943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:41609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:20:05.046624 11536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:20:05.048013 11536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:20:05.060817 11541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:20:05.060787 11542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:20:05.061043 11544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:20:05.061599 11536 server_base.cc:1034] running on GCE node
I20260929 20:20:05.062358 11536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:20:05.063556 11536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:20:05.064774 11536 hybrid_clock.cc:648] HybridClock initialized: now 1790713205064722 us; error 43 us; skew 500 ppm
I20260929 20:20:05.065135 11536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:20:05 dist-test-slave-4cx3 krb5kdc[11430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713205, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:20:05.068454 11536 init.cc:377] Logged in from keytab as kudu/127.27.185.65@KRBTEST.COM (short username kudu)
I20260929 20:20:05.069985 11536 webserver.cc:460] Webserver started at http://127.27.185.65:39177/ using document root <none> and password file <none>
I20260929 20:20:05.070695 11536 fs_manager.cc:359] Metadata directory not provided
I20260929 20:20:05.070847 11536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:20:05.071164 11536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:20:05.074333 11536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/data/instance:
uuid: "1eaa1e9d171d400e913daf376fa7943f"
format_stamp: "Formatted at 2026-09-29 20:20:05 on dist-test-slave-4cx3"
server_key: "3f1f32d5303ba34e16b76822fd6592af"
server_key_iv: "e2f9e954fdfc11fd81a9650954cbc072"
server_key_version: "encryptionkey@0"
I20260929 20:20:05.075130 11536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance:
uuid: "1eaa1e9d171d400e913daf376fa7943f"
format_stamp: "Formatted at 2026-09-29 20:20:05 on dist-test-slave-4cx3"
server_key: "3f1f32d5303ba34e16b76822fd6592af"
server_key_iv: "e2f9e954fdfc11fd81a9650954cbc072"
server_key_version: "encryptionkey@0"
I20260929 20:20:05.080860 11536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260929 20:20:05.084640 11551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:20:05.086109 11536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 20:20:05.086310 11536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "1eaa1e9d171d400e913daf376fa7943f"
format_stamp: "Formatted at 2026-09-29 20:20:05 on dist-test-slave-4cx3"
server_key: "3f1f32d5303ba34e16b76822fd6592af"
server_key_iv: "e2f9e954fdfc11fd81a9650954cbc072"
server_key_version: "encryptionkey@0"
I20260929 20:20:05.086540 11536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:20:05.098047 11536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:20:05.102003 11536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:20:05.102301 11536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:20:05.103235 11536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:20:05.104607 11536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:20:05.104708 11536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:20:05.104862 11536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:20:05.104934 11536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:20:05.138720 11536 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:42891
I20260929 20:20:05.138761 11663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:42891 every 8 connection(s)
I20260929 20:20:05.140161 11536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:20:05.147109 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 11536
I20260929 20:20:05.147326 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance
I20260929 20:20:05.148133 28389 external_mini_cluster.cc:1428] Setting key 153518ff1a1189643c9d4208d74fb885
Sep 29 20:20:05 dist-test-slave-4cx3 krb5kdc[11430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713205, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:20:05.162220 11666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:05.142680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:36489 (local address 127.27.185.126:41609)
0929 20:20:05.143150 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:05.143154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:05.144139 (+ 985us) server_negotiation.cc:407] Connection header received
0929 20:20:05.146038 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:05.146044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:05.146152 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:05.146273 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:20:05.148392 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:05.149342 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:05.151722 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:05.152111 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:05.152207 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:05.156429 (+ 4222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:20:05.156434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:20:05.156437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:20:05.156467 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:20:05.159067 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:05.159865 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:05.159875 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:05.159877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:05.159995 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:05.160443 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:05.160448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:05.160451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:05.160720 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:20:05.160878 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:05.161613 (+ 735us) server_negotiation.cc:299] Negotiation successful
0929 20:20:05.161845 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":257,"threads_started":1}
I20260929 20:20:05.163240 11667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:05.143212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.65:36489)
0929 20:20:05.143962 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:05.144000 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 20:20:05.145758 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:20:05.146527 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:20:05.146536 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:20:05.147343 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:20:05.148204 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:05.148233 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:05.149511 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:05.149515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:05.151563 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:05.151588 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:05.152353 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:05.152420 (+ 67us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:05.152744 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:05.153486 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:20:05.153519 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:20:05.156196 (+ 2677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:20:05.159207 (+ 3011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:20:05.159216 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:20:05.159234 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:20:05.159739 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:20:05.160101 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:20:05.160109 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:20:05.160113 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:20:05.160323 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:20:05.160850 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:20:05.160858 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:20:05.161216 (+ 358us) client_negotiation.cc:769] Sending connection context
0929 20:20:05.161484 (+ 268us) client_negotiation.cc:240] Negotiation successful
0929 20:20:05.162044 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":304,"threads_started":1}
I20260929 20:20:05.164870 11664 heartbeater.cc:347] Connected to a master server at 127.27.185.126:41609
I20260929 20:20:05.165200 11664 heartbeater.cc:464] Registering TS with master...
I20260929 20:20:05.165975 11664 heartbeater.cc:511] Master 127.27.185.126:41609 requested a full tablet report, sending...
I20260929 20:20:05.168227 11475 ts_manager.cc:194] Registered new tserver with Master: 1eaa1e9d171d400e913daf376fa7943f (127.27.185.65:42891)
I20260929 20:20:05.169711 11475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.65@KRBTEST.COM'} at 127.27.185.65:36489
WARNING: no policy specified for kudu/127.27.185.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
I20260929 20:20:05.202231 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:0
--local_ip_for_outbound_sockets=127.27.185.66
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:41609
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:43943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:20:05.537937 11670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:20:05.538282 11670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:20:05.538430 11670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:20:05.546123 11670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:20:05.546239 11670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:20:05.546458 11670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:20:05.556329 11670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:43943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:41609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:20:05.557683 11670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:20:05.559008 11670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:20:05.572324 11675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:20:05.572345 11676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:20:05.572293 11678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:20:05.573349 11670 server_base.cc:1034] running on GCE node
I20260929 20:20:05.574064 11670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:20:05.575080 11670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:20:05.576346 11670 hybrid_clock.cc:648] HybridClock initialized: now 1790713205576292 us; error 44 us; skew 500 ppm
I20260929 20:20:05.576726 11670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:20:05 dist-test-slave-4cx3 krb5kdc[11430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713205, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:20:05.580219 11670 init.cc:377] Logged in from keytab as kudu/127.27.185.66@KRBTEST.COM (short username kudu)
I20260929 20:20:05.581768 11670 webserver.cc:460] Webserver started at http://127.27.185.66:45655/ using document root <none> and password file <none>
I20260929 20:20:05.582504 11670 fs_manager.cc:359] Metadata directory not provided
I20260929 20:20:05.582655 11670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:20:05.582965 11670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:20:05.586017 11670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/data/instance:
uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a"
format_stamp: "Formatted at 2026-09-29 20:20:05 on dist-test-slave-4cx3"
server_key: "67af568a608460a635d7b4363445ee6a"
server_key_iv: "d3666038e879091cc8912c746c0d68c2"
server_key_version: "encryptionkey@0"
I20260929 20:20:05.586856 11670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance:
uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a"
format_stamp: "Formatted at 2026-09-29 20:20:05 on dist-test-slave-4cx3"
server_key: "67af568a608460a635d7b4363445ee6a"
server_key_iv: "d3666038e879091cc8912c746c0d68c2"
server_key_version: "encryptionkey@0"
I20260929 20:20:05.592648 11670 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 20:20:05.596223 11685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:20:05.597568 11670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:20:05.597756 11670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a"
format_stamp: "Formatted at 2026-09-29 20:20:05 on dist-test-slave-4cx3"
server_key: "67af568a608460a635d7b4363445ee6a"
server_key_iv: "d3666038e879091cc8912c746c0d68c2"
server_key_version: "encryptionkey@0"
I20260929 20:20:05.597960 11670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:20:05.617187 11670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:20:05.621618 11670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:20:05.621930 11670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:20:05.622963 11670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:20:05.624437 11670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:20:05.624552 11670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:20:05.624706 11670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:20:05.624778 11670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:20:05.661207 11670 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:38641
I20260929 20:20:05.661247 11797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:38641 every 8 connection(s)
I20260929 20:20:05.662559 11670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:20:05.665700 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 11670
I20260929 20:20:05.665911 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance
I20260929 20:20:05.666590 28389 external_mini_cluster.cc:1428] Setting key 4d857ca04aae4a8c1ffd9e1c1e6fc440
Sep 29 20:20:05 dist-test-slave-4cx3 krb5kdc[11430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713205, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:20:05.686908 11800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:05.665133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:48571 (local address 127.27.185.126:41609)
0929 20:20:05.665609 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:05.665616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:05.666958 (+ 1342us) server_negotiation.cc:407] Connection header received
0929 20:20:05.669299 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:05.669303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:05.669376 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:05.669467 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:20:05.671650 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:05.672387 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:05.674955 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:05.675673 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:05.675770 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:05.679640 (+ 3870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:20:05.679649 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:20:05.679658 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:20:05.679703 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:20:05.683271 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:05.684083 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:05.684092 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:05.684098 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:05.684258 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:05.684679 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:05.684686 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:05.684692 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:05.685086 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:20:05.685242 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:05.686190 (+ 948us) server_negotiation.cc:299] Negotiation successful
0929 20:20:05.686535 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":233,"threads_started":1}
I20260929 20:20:05.687929 11801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:05.665759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.66:48571)
0929 20:20:05.666753 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:05.666807 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 20:20:05.668996 (+ 2189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:20:05.669761 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:20:05.669775 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:20:05.670593 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:20:05.671472 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:05.671501 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:05.672558 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:05.672567 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:05.674798 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:05.674816 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:05.675899 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:05.675909 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:05.676242 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:05.676982 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:20:05.677022 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:20:05.679346 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:20:05.683479 (+ 4133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:20:05.683486 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:20:05.683500 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:20:05.683930 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:20:05.684368 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:20:05.684371 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:20:05.684373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:20:05.684561 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:20:05.685205 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:20:05.685211 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:20:05.685630 (+ 419us) client_negotiation.cc:769] Sending connection context
0929 20:20:05.686032 (+ 402us) client_negotiation.cc:240] Negotiation successful
0929 20:20:05.686581 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":361,"threads_started":1}
I20260929 20:20:05.689508 11798 heartbeater.cc:347] Connected to a master server at 127.27.185.126:41609
I20260929 20:20:05.689853 11798 heartbeater.cc:464] Registering TS with master...
I20260929 20:20:05.690702 11798 heartbeater.cc:511] Master 127.27.185.126:41609 requested a full tablet report, sending...
I20260929 20:20:05.692284 11475 ts_manager.cc:194] Registered new tserver with Master: 0e4a8bb2f91540cd9e2c15bb3408f43a (127.27.185.66:38641)
I20260929 20:20:05.693110 11475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.66@KRBTEST.COM'} at 127.27.185.66:48571
WARNING: no policy specified for kudu/127.27.185.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
I20260929 20:20:05.728099 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:0
--local_ip_for_outbound_sockets=127.27.185.67
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:41609
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:43943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:20:06.068960 11804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:20:06.069295 11804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:20:06.069479 11804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:20:06.077037 11804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:20:06.077200 11804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:20:06.077385 11804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:20:06.087747 11804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:43943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:41609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:20:06.089237 11804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:20:06.090750 11804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:20:06.103798 11809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:20:06.104400 11810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:20:06.104470 11812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:20:06.105263 11804 server_base.cc:1034] running on GCE node
I20260929 20:20:06.105805 11804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:20:06.106729 11804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:20:06.107923 11804 hybrid_clock.cc:648] HybridClock initialized: now 1790713206107870 us; error 41 us; skew 500 ppm
I20260929 20:20:06.108214 11804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:20:06 dist-test-slave-4cx3 krb5kdc[11430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713206, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:20:06.111358 11804 init.cc:377] Logged in from keytab as kudu/127.27.185.67@KRBTEST.COM (short username kudu)
I20260929 20:20:06.112886 11804 webserver.cc:460] Webserver started at http://127.27.185.67:35807/ using document root <none> and password file <none>
I20260929 20:20:06.113586 11804 fs_manager.cc:359] Metadata directory not provided
I20260929 20:20:06.113724 11804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:20:06.114039 11804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:20:06.116978 11804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/data/instance:
uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2"
format_stamp: "Formatted at 2026-09-29 20:20:06 on dist-test-slave-4cx3"
server_key: "5aebf81f95b9b81b6ed0a11724b8569a"
server_key_iv: "fcb66ae6f8487cbc9808933eea0c13a0"
server_key_version: "encryptionkey@0"
I20260929 20:20:06.117736 11804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance:
uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2"
format_stamp: "Formatted at 2026-09-29 20:20:06 on dist-test-slave-4cx3"
server_key: "5aebf81f95b9b81b6ed0a11724b8569a"
server_key_iv: "fcb66ae6f8487cbc9808933eea0c13a0"
server_key_version: "encryptionkey@0"
I20260929 20:20:06.122769 11804 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 20:20:06.126144 11819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:20:06.127552 11804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:20:06.127722 11804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2"
format_stamp: "Formatted at 2026-09-29 20:20:06 on dist-test-slave-4cx3"
server_key: "5aebf81f95b9b81b6ed0a11724b8569a"
server_key_iv: "fcb66ae6f8487cbc9808933eea0c13a0"
server_key_version: "encryptionkey@0"
I20260929 20:20:06.127913 11804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:20:06.141906 11804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:20:06.146234 11804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:20:06.146584 11804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:20:06.147620 11804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:20:06.148912 11804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:20:06.149019 11804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:20:06.149173 11804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:20:06.149250 11804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:20:06.172750 11664 heartbeater.cc:503] Master 127.27.185.126:41609 was elected leader, sending a full tablet report...
I20260929 20:20:06.184816 11804 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:45957
I20260929 20:20:06.184854 11931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:45957 every 8 connection(s)
I20260929 20:20:06.186147 11804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:20:06.192768 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 11804
I20260929 20:20:06.192996 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance
I20260929 20:20:06.193572 28389 external_mini_cluster.cc:1428] Setting key 70c1d235bf93923144fa8b3d0e927cb0
Sep 29 20:20:06 dist-test-slave-4cx3 krb5kdc[11430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713206, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:20:06.207643 11934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:06.188998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:51325 (local address 127.27.185.126:41609)
0929 20:20:06.189446 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:06.189451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:06.190590 (+ 1139us) server_negotiation.cc:407] Connection header received
0929 20:20:06.192665 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:06.192669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:06.192739 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:06.192834 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:20:06.194860 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.196000 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.197774 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.198156 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.198251 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:06.201577 (+ 3326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:20:06.201582 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:20:06.201585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:20:06.201617 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:20:06.204791 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:06.205451 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:06.205454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:06.205456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:06.205584 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:06.205961 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:06.205964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:06.205966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:06.206212 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:20:06.206328 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:06.206933 (+ 605us) server_negotiation.cc:299] Negotiation successful
0929 20:20:06.207165 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":206,"threads_started":1}
I20260929 20:20:06.208547 11935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:06.189532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.67:51325)
0929 20:20:06.190418 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:06.190460 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:20:06.192301 (+ 1841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:20:06.193140 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:20:06.193154 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:20:06.193886 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:20:06.194691 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:06.194710 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.196153 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.196157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:06.197645 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:06.197657 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.198332 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.198336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:06.198565 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:06.199159 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:20:06.199180 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:20:06.201378 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:20:06.204929 (+ 3551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:20:06.204935 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:20:06.204948 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:20:06.205346 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:20:06.205677 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:20:06.205680 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:20:06.205682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:20:06.205855 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:20:06.206313 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:20:06.206319 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:20:06.206623 (+ 304us) client_negotiation.cc:769] Sending connection context
0929 20:20:06.206844 (+ 221us) client_negotiation.cc:240] Negotiation successful
0929 20:20:06.207250 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":311,"threads_started":1}
I20260929 20:20:06.210050 11932 heartbeater.cc:347] Connected to a master server at 127.27.185.126:41609
I20260929 20:20:06.210305 11932 heartbeater.cc:464] Registering TS with master...
I20260929 20:20:06.211002 11932 heartbeater.cc:511] Master 127.27.185.126:41609 requested a full tablet report, sending...
I20260929 20:20:06.212491 11475 ts_manager.cc:194] Registered new tserver with Master: ccce56dac1a84ed391bdc11b7dd6a1d2 (127.27.185.67:45957)
I20260929 20:20:06.213253 11475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.67@KRBTEST.COM'} at 127.27.185.67:51325
I20260929 20:20:06.221565 28389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:20:06.239346 11934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:06.225942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33696 (local address 127.27.185.126:41609)
0929 20:20:06.226124 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:06.226128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:06.226759 (+ 631us) server_negotiation.cc:407] Connection header received
0929 20:20:06.226821 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:06.226824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:06.226892 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:06.226982 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:20:06.228593 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.229346 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.230846 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.231301 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.231438 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:06.233992 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:20:06.233997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:20:06.234000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:20:06.234032 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:20:06.236711 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:06.237353 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:06.237358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:06.237362 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:06.237520 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:06.237894 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:06.237899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:06.237903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:06.238214 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:20:06.238469 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:06.238781 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 20:20:06.238997 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:20:06.243235 11475 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:20:06.250171 11475 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:20:06.289418 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:06.261977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:42891 (local address 127.0.0.1:59916)
0929 20:20:06.277483 (+ 15506us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:06.277538 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 20:20:06.277767 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:20:06.279479 (+ 1712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:20:06.279489 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:20:06.279549 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:20:06.281435 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:06.281451 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.283357 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.283362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:06.285959 (+ 2597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:06.285973 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.288238 (+ 2265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.288246 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:06.288511 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:06.288569 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 20:20:06.288695 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 20:20:06.288904 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15240,"mutex_wait_us":59,"thread_start_us":18571,"threads_started":1}
I20260929 20:20:06.289449 11946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:06.262207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59916 (local address 127.27.185.65:42891)
0929 20:20:06.274852 (+ 12645us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:06.274867 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:06.278054 (+ 3187us) server_negotiation.cc:407] Connection header received
0929 20:20:06.278198 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:06.278210 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:06.278573 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:06.278814 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:20:06.281693 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.283208 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.286134 (+ 2926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.287709 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.287846 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:06.288086 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:06.288914 (+ 828us) server_negotiation.cc:299] Negotiation successful
0929 20:20:06.289086 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12410,"thread_start_us":12237,"threads_started":1}
I20260929 20:20:06.289512 11949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:06.262974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56700 (local address 127.27.185.66:38641)
0929 20:20:06.278932 (+ 15958us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:06.278945 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:06.281788 (+ 2843us) server_negotiation.cc:407] Connection header received
0929 20:20:06.281899 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:06.281905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:06.282155 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:06.282346 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:20:06.282977 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.284357 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.286812 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.287963 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.288089 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:06.288297 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:06.289018 (+ 721us) server_negotiation.cc:299] Negotiation successful
0929 20:20:06.289162 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15733,"thread_start_us":15976,"threads_started":1}
I20260929 20:20:06.290060 11950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:06.262897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42358 (local address 127.27.185.67:45957)
0929 20:20:06.276833 (+ 13936us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:06.276844 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:06.281118 (+ 4274us) server_negotiation.cc:407] Connection header received
0929 20:20:06.281422 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:06.281433 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:06.281694 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:06.281891 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:20:06.282549 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.284092 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.286861 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.288314 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.288434 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:06.288664 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:06.289535 (+ 871us) server_negotiation.cc:299] Negotiation successful
0929 20:20:06.289729 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13689,"thread_start_us":16189,"threads_started":1}
I20260929 20:20:06.290102 11948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:06.262771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:38641 (local address 127.0.0.1:56700)
0929 20:20:06.281527 (+ 18756us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:06.281545 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:20:06.281658 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:20:06.282351 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:20:06.282356 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:20:06.282386 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:20:06.282811 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:06.282822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.284558 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.284570 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:06.286640 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:06.286657 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.288431 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.288437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:06.288635 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:06.288673 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 20:20:06.288769 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 20:20:06.288914 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18618,"mutex_wait_us":166,"thread_start_us":18424,"threads_started":1}
I20260929 20:20:06.290460 11945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:06.262356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:45957 (local address 127.0.0.1:42358)
0929 20:20:06.280957 (+ 18601us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:06.280986 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 20:20:06.281162 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:20:06.281915 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:20:06.281919 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:20:06.281944 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:20:06.282302 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:06.282318 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.284265 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.284270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:06.286697 (+ 2427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:06.286709 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.288805 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.288810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:06.289006 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:06.289051 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 20:20:06.289137 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 20:20:06.289258 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18425,"mutex_wait_us":27,"thread_start_us":14713,"threads_started":1}
I20260929 20:20:06.293886 11733 tablet_service.cc:1467] Processing CreateTablet for tablet 01dd37bf30b64aad9061b3d41ecbb75b (DEFAULT_TABLE table=test-table [id=f77feeaf98e246b6bdfde0db0c197438]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:20:06.293994 11867 tablet_service.cc:1467] Processing CreateTablet for tablet 01dd37bf30b64aad9061b3d41ecbb75b (DEFAULT_TABLE table=test-table [id=f77feeaf98e246b6bdfde0db0c197438]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:20:06.293995 11599 tablet_service.cc:1467] Processing CreateTablet for tablet 01dd37bf30b64aad9061b3d41ecbb75b (DEFAULT_TABLE table=test-table [id=f77feeaf98e246b6bdfde0db0c197438]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:20:06.295131 11733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01dd37bf30b64aad9061b3d41ecbb75b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:20:06.295374 11867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01dd37bf30b64aad9061b3d41ecbb75b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:20:06.295374 11599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01dd37bf30b64aad9061b3d41ecbb75b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:20:06.307056 11951 tablet_bootstrap.cc:492] T 01dd37bf30b64aad9061b3d41ecbb75b P 0e4a8bb2f91540cd9e2c15bb3408f43a: Bootstrap starting.
I20260929 20:20:06.309827 11952 tablet_bootstrap.cc:492] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f: Bootstrap starting.
I20260929 20:20:06.311540 11953 tablet_bootstrap.cc:492] T 01dd37bf30b64aad9061b3d41ecbb75b P ccce56dac1a84ed391bdc11b7dd6a1d2: Bootstrap starting.
I20260929 20:20:06.312114 11951 tablet_bootstrap.cc:654] T 01dd37bf30b64aad9061b3d41ecbb75b P 0e4a8bb2f91540cd9e2c15bb3408f43a: Neither blocks nor log segments found. Creating new log.
I20260929 20:20:06.313474 11952 tablet_bootstrap.cc:654] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f: Neither blocks nor log segments found. Creating new log.
I20260929 20:20:06.313522 11951 log.cc:824] T 01dd37bf30b64aad9061b3d41ecbb75b P 0e4a8bb2f91540cd9e2c15bb3408f43a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:20:06.314566 11953 tablet_bootstrap.cc:654] T 01dd37bf30b64aad9061b3d41ecbb75b P ccce56dac1a84ed391bdc11b7dd6a1d2: Neither blocks nor log segments found. Creating new log.
I20260929 20:20:06.314628 11952 log.cc:824] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:20:06.315757 11953 log.cc:824] T 01dd37bf30b64aad9061b3d41ecbb75b P ccce56dac1a84ed391bdc11b7dd6a1d2: Log is configured to *not* fsync() on all Append() calls
I20260929 20:20:06.316728 11951 tablet_bootstrap.cc:492] T 01dd37bf30b64aad9061b3d41ecbb75b P 0e4a8bb2f91540cd9e2c15bb3408f43a: No bootstrap required, opened a new log
I20260929 20:20:06.317040 11951 ts_tablet_manager.cc:1397] T 01dd37bf30b64aad9061b3d41ecbb75b P 0e4a8bb2f91540cd9e2c15bb3408f43a: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260929 20:20:06.317709 11952 tablet_bootstrap.cc:492] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f: No bootstrap required, opened a new log
I20260929 20:20:06.317972 11952 ts_tablet_manager.cc:1397] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260929 20:20:06.318337 11953 tablet_bootstrap.cc:492] T 01dd37bf30b64aad9061b3d41ecbb75b P ccce56dac1a84ed391bdc11b7dd6a1d2: No bootstrap required, opened a new log
I20260929 20:20:06.318588 11953 ts_tablet_manager.cc:1397] T 01dd37bf30b64aad9061b3d41ecbb75b P ccce56dac1a84ed391bdc11b7dd6a1d2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 20:20:06.326102 11953 raft_consensus.cc:348] T 01dd37bf30b64aad9061b3d41ecbb75b P ccce56dac1a84ed391bdc11b7dd6a1d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eaa1e9d171d400e913daf376fa7943f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42891 } } peers { permanent_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45957 } } peers { permanent_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38641 } }
I20260929 20:20:06.326475 11953 raft_consensus.cc:374] T 01dd37bf30b64aad9061b3d41ecbb75b P ccce56dac1a84ed391bdc11b7dd6a1d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:20:06.326576 11953 raft_consensus.cc:729] T 01dd37bf30b64aad9061b3d41ecbb75b P ccce56dac1a84ed391bdc11b7dd6a1d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccce56dac1a84ed391bdc11b7dd6a1d2, State: Initialized, Role: FOLLOWER
I20260929 20:20:06.327204 11953 consensus_queue.cc:260] T 01dd37bf30b64aad9061b3d41ecbb75b P ccce56dac1a84ed391bdc11b7dd6a1d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eaa1e9d171d400e913daf376fa7943f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42891 } } peers { permanent_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45957 } } peers { permanent_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38641 } }
I20260929 20:20:06.328547 11952 raft_consensus.cc:348] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eaa1e9d171d400e913daf376fa7943f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42891 } } peers { permanent_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45957 } } peers { permanent_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38641 } }
I20260929 20:20:06.328549 11951 raft_consensus.cc:348] T 01dd37bf30b64aad9061b3d41ecbb75b P 0e4a8bb2f91540cd9e2c15bb3408f43a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eaa1e9d171d400e913daf376fa7943f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42891 } } peers { permanent_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45957 } } peers { permanent_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38641 } }
I20260929 20:20:06.328965 11951 raft_consensus.cc:374] T 01dd37bf30b64aad9061b3d41ecbb75b P 0e4a8bb2f91540cd9e2c15bb3408f43a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:20:06.328969 11952 raft_consensus.cc:374] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:20:06.329071 11952 raft_consensus.cc:729] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eaa1e9d171d400e913daf376fa7943f, State: Initialized, Role: FOLLOWER
I20260929 20:20:06.329152 11951 raft_consensus.cc:729] T 01dd37bf30b64aad9061b3d41ecbb75b P 0e4a8bb2f91540cd9e2c15bb3408f43a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4a8bb2f91540cd9e2c15bb3408f43a, State: Initialized, Role: FOLLOWER
I20260929 20:20:06.329682 11932 heartbeater.cc:503] Master 127.27.185.126:41609 was elected leader, sending a full tablet report...
I20260929 20:20:06.329824 11951 consensus_queue.cc:260] T 01dd37bf30b64aad9061b3d41ecbb75b P 0e4a8bb2f91540cd9e2c15bb3408f43a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eaa1e9d171d400e913daf376fa7943f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42891 } } peers { permanent_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45957 } } peers { permanent_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38641 } }
I20260929 20:20:06.329824 11952 consensus_queue.cc:260] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eaa1e9d171d400e913daf376fa7943f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42891 } } peers { permanent_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45957 } } peers { permanent_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38641 } }
I20260929 20:20:06.331761 11953 ts_tablet_manager.cc:1428] T 01dd37bf30b64aad9061b3d41ecbb75b P ccce56dac1a84ed391bdc11b7dd6a1d2: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260929 20:20:06.338367 11952 ts_tablet_manager.cc:1428] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f: Time spent starting tablet: real 0.020s user 0.015s sys 0.006s
I20260929 20:20:06.340857 11798 heartbeater.cc:503] Master 127.27.185.126:41609 was elected leader, sending a full tablet report...
I20260929 20:20:06.341699 11951 ts_tablet_manager.cc:1428] T 01dd37bf30b64aad9061b3d41ecbb75b P 0e4a8bb2f91540cd9e2c15bb3408f43a: Time spent starting tablet: real 0.025s user 0.016s sys 0.004s
W20260929 20:20:06.392542 11665 tablet.cc:2369] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:20:06.414736 11799 tablet.cc:2369] T 01dd37bf30b64aad9061b3d41ecbb75b P 0e4a8bb2f91540cd9e2c15bb3408f43a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:20:06.437834 11933 tablet.cc:2369] T 01dd37bf30b64aad9061b3d41ecbb75b P ccce56dac1a84ed391bdc11b7dd6a1d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:20:06.438364 11958 raft_consensus.cc:482] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:20:06.438687 11958 raft_consensus.cc:504] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eaa1e9d171d400e913daf376fa7943f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42891 } } peers { permanent_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45957 } } peers { permanent_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38641 } }
I20260929 20:20:06.440794 11958 leader_election.cc:288] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccce56dac1a84ed391bdc11b7dd6a1d2 (127.27.185.67:45957), 0e4a8bb2f91540cd9e2c15bb3408f43a (127.27.185.66:38641)
I20260929 20:20:06.448045 11950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:06.441204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:35361 (local address 127.27.185.67:45957)
0929 20:20:06.441410 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:06.441416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:06.442228 (+ 812us) server_negotiation.cc:407] Connection header received
0929 20:20:06.442390 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:06.442395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:06.442478 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:06.442609 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:20:06.443287 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.444175 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.446098 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.446886 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.447044 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:06.447131 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:06.447632 (+ 501us) server_negotiation.cc:299] Negotiation successful
0929 20:20:06.447761 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:20:06.448046 11949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:06.441321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:44339 (local address 127.27.185.66:38641)
0929 20:20:06.441564 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:06.441573 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:06.442384 (+ 811us) server_negotiation.cc:407] Connection header received
0929 20:20:06.442549 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:06.442554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:06.442631 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:06.442732 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:20:06.443484 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.444192 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.446175 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.446930 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.447123 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:06.447191 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:06.447633 (+ 442us) server_negotiation.cc:299] Negotiation successful
0929 20:20:06.447760 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:20:06.448242 11961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:06.441460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:38641 (local address 127.27.185.65:44339)
0929 20:20:06.442230 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:06.442255 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 20:20:06.442407 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:20:06.442804 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:20:06.442812 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:20:06.442853 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:20:06.443263 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:06.443276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.444357 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.444361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:06.445961 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:06.445974 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.447193 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.447199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:06.447323 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:06.447491 (+ 168us) client_negotiation.cc:769] Sending connection context
0929 20:20:06.447596 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 20:20:06.447739 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"mutex_wait_us":221,"thread_start_us":407,"threads_started":1}
I20260929 20:20:06.448266 11960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:06.441250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:45957 (local address 127.27.185.65:35361)
0929 20:20:06.442101 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:06.442127 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 20:20:06.442250 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:20:06.442580 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:20:06.442584 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:20:06.442606 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:20:06.443128 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:06.443140 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.444374 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.444379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:06.445923 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:06.445936 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.447130 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.447136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:06.447303 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:06.447489 (+ 186us) client_negotiation.cc:769] Sending connection context
0929 20:20:06.447601 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 20:20:06.447741 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"mutex_wait_us":8,"spinlock_wait_cycles":36608,"thread_start_us":451,"threads_started":1}
I20260929 20:20:06.449383 11753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01dd37bf30b64aad9061b3d41ecbb75b" candidate_uuid: "1eaa1e9d171d400e913daf376fa7943f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a" is_pre_election: true
I20260929 20:20:06.449442 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01dd37bf30b64aad9061b3d41ecbb75b" candidate_uuid: "1eaa1e9d171d400e913daf376fa7943f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" is_pre_election: true
I20260929 20:20:06.449918 11753 raft_consensus.cc:2454] T 01dd37bf30b64aad9061b3d41ecbb75b P 0e4a8bb2f91540cd9e2c15bb3408f43a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eaa1e9d171d400e913daf376fa7943f in term 0.
I20260929 20:20:06.449941 11887 raft_consensus.cc:2454] T 01dd37bf30b64aad9061b3d41ecbb75b P ccce56dac1a84ed391bdc11b7dd6a1d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eaa1e9d171d400e913daf376fa7943f in term 0.
I20260929 20:20:06.450613 11555 leader_election.cc:302] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0e4a8bb2f91540cd9e2c15bb3408f43a, 1eaa1e9d171d400e913daf376fa7943f, ccce56dac1a84ed391bdc11b7dd6a1d2; no voters:
I20260929 20:20:06.451037 11958 raft_consensus.cc:2781] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:20:06.451159 11958 raft_consensus.cc:482] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:20:06.451299 11958 raft_consensus.cc:3037] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:20:06.453487 11958 raft_consensus.cc:504] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eaa1e9d171d400e913daf376fa7943f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42891 } } peers { permanent_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45957 } } peers { permanent_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38641 } }
I20260929 20:20:06.454116 11958 leader_election.cc:288] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f [CANDIDATE]: Term 1 election: Requested vote from peers ccce56dac1a84ed391bdc11b7dd6a1d2 (127.27.185.67:45957), 0e4a8bb2f91540cd9e2c15bb3408f43a (127.27.185.66:38641)
I20260929 20:20:06.454636 11753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01dd37bf30b64aad9061b3d41ecbb75b" candidate_uuid: "1eaa1e9d171d400e913daf376fa7943f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a"
I20260929 20:20:06.454614 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01dd37bf30b64aad9061b3d41ecbb75b" candidate_uuid: "1eaa1e9d171d400e913daf376fa7943f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2"
I20260929 20:20:06.454831 11887 raft_consensus.cc:3037] T 01dd37bf30b64aad9061b3d41ecbb75b P ccce56dac1a84ed391bdc11b7dd6a1d2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:20:06.454828 11753 raft_consensus.cc:3037] T 01dd37bf30b64aad9061b3d41ecbb75b P 0e4a8bb2f91540cd9e2c15bb3408f43a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:20:06.458019 11887 raft_consensus.cc:2454] T 01dd37bf30b64aad9061b3d41ecbb75b P ccce56dac1a84ed391bdc11b7dd6a1d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eaa1e9d171d400e913daf376fa7943f in term 1.
I20260929 20:20:06.458019 11753 raft_consensus.cc:2454] T 01dd37bf30b64aad9061b3d41ecbb75b P 0e4a8bb2f91540cd9e2c15bb3408f43a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eaa1e9d171d400e913daf376fa7943f in term 1.
I20260929 20:20:06.458434 11554 leader_election.cc:302] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4a8bb2f91540cd9e2c15bb3408f43a, 1eaa1e9d171d400e913daf376fa7943f; no voters:
I20260929 20:20:06.458792 11958 raft_consensus.cc:2781] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:20:06.459246 11958 raft_consensus.cc:686] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f [term 1 LEADER]: Becoming Leader. State: Replica: 1eaa1e9d171d400e913daf376fa7943f, State: Running, Role: LEADER
I20260929 20:20:06.459798 11958 consensus_queue.cc:237] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eaa1e9d171d400e913daf376fa7943f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42891 } } peers { permanent_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45957 } } peers { permanent_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38641 } }
I20260929 20:20:06.464641 11475 catalog_manager.cc:5495] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eaa1e9d171d400e913daf376fa7943f (127.27.185.65). New cstate: current_term: 1 leader_uuid: "1eaa1e9d171d400e913daf376fa7943f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eaa1e9d171d400e913daf376fa7943f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38641 } health_report { overall_health: UNKNOWN } } }
I20260929 20:20:06.484180 11946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:06.478661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59926 (local address 127.27.185.65:42891)
0929 20:20:06.478873 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:06.478879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:06.478897 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:20:06.479133 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:06.479143 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:06.479234 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:06.479389 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:20:06.479819 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.480563 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.482015 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.482407 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.482541 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:06.483052 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:20:06.483620 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:20:06.483780 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:06.483833 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 20:20:06.483912 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:20:06.492713 11887 raft_consensus.cc:1261] T 01dd37bf30b64aad9061b3d41ecbb75b P ccce56dac1a84ed391bdc11b7dd6a1d2 [term 1 FOLLOWER]: Refusing update from remote peer 1eaa1e9d171d400e913daf376fa7943f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:20:06.492738 11753 raft_consensus.cc:1261] T 01dd37bf30b64aad9061b3d41ecbb75b P 0e4a8bb2f91540cd9e2c15bb3408f43a [term 1 FOLLOWER]: Refusing update from remote peer 1eaa1e9d171d400e913daf376fa7943f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:20:06.493695 11962 consensus_queue.cc:1035] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:20:06.494073 11958 consensus_queue.cc:1035] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:20:06.512830 11965 mvcc.cc:205] Tried to move back new op lower bound from 7334761293783212032 to 7334761293663186944. Current Snapshot: MvccSnapshot[applied={T|T < 7334761293783212032}]
I20260929 20:20:06.538285 11475 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:20:06.538582 11475 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:20:06.542526 11475 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134: Sending DeleteTablet for 3 replicas of tablet 01dd37bf30b64aad9061b3d41ecbb75b
I20260929 20:20:06.543783 11599 tablet_service.cc:1514] Processing DeleteTablet for tablet 01dd37bf30b64aad9061b3d41ecbb75b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:20:06 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:59916
I20260929 20:20:06.543783 11867 tablet_service.cc:1514] Processing DeleteTablet for tablet 01dd37bf30b64aad9061b3d41ecbb75b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:20:06 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:42358
I20260929 20:20:06.544147 11733 tablet_service.cc:1514] Processing DeleteTablet for tablet 01dd37bf30b64aad9061b3d41ecbb75b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:20:06 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:56700
I20260929 20:20:06.544854 11976 tablet_replica.cc:330] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f: stopping tablet replica
I20260929 20:20:06.545147 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 11536
I20260929 20:20:06.545195 11975 tablet_replica.cc:330] T 01dd37bf30b64aad9061b3d41ecbb75b P ccce56dac1a84ed391bdc11b7dd6a1d2: stopping tablet replica
I20260929 20:20:06.545218 11976 raft_consensus.cc:2229] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:20:06.545590 11975 raft_consensus.cc:2229] T 01dd37bf30b64aad9061b3d41ecbb75b P ccce56dac1a84ed391bdc11b7dd6a1d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:20:06.545955 11976 raft_consensus.cc:2258] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:20:06.546106 11975 raft_consensus.cc:2258] T 01dd37bf30b64aad9061b3d41ecbb75b P ccce56dac1a84ed391bdc11b7dd6a1d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:20:06.548229 11976 ts_tablet_manager.cc:1905] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:20:06.556102 11976 ts_tablet_manager.cc:1918] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:20:06.556857 11977 tablet_replica.cc:330] T 01dd37bf30b64aad9061b3d41ecbb75b P 0e4a8bb2f91540cd9e2c15bb3408f43a: stopping tablet replica
I20260929 20:20:06.557356 11977 raft_consensus.cc:2229] T 01dd37bf30b64aad9061b3d41ecbb75b P 0e4a8bb2f91540cd9e2c15bb3408f43a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:20:06.557936 11977 raft_consensus.cc:2258] T 01dd37bf30b64aad9061b3d41ecbb75b P 0e4a8bb2f91540cd9e2c15bb3408f43a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:20:06.558426 11976 log.cc:1196] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/wal/wals/01dd37bf30b64aad9061b3d41ecbb75b
I20260929 20:20:06.559162 11976 ts_tablet_manager.cc:1939] T 01dd37bf30b64aad9061b3d41ecbb75b P 1eaa1e9d171d400e913daf376fa7943f: Deleting consensus metadata
I20260929 20:20:06.560593 11977 ts_tablet_manager.cc:1905] T 01dd37bf30b64aad9061b3d41ecbb75b P 0e4a8bb2f91540cd9e2c15bb3408f43a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:20:06.560761 11975 ts_tablet_manager.cc:1905] T 01dd37bf30b64aad9061b3d41ecbb75b P ccce56dac1a84ed391bdc11b7dd6a1d2: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 20:20:06 dist-test-slave-4cx3 krb5kdc[11430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.65@KRBTEST.COM
I20260929 20:20:06.561218 11459 catalog_manager.cc:4841] TS 1eaa1e9d171d400e913daf376fa7943f (127.27.185.65:42891): tablet 01dd37bf30b64aad9061b3d41ecbb75b (table test-table [id=f77feeaf98e246b6bdfde0db0c197438]) successfully deleted
I20260929 20:20:06.566346 11977 ts_tablet_manager.cc:1918] T 01dd37bf30b64aad9061b3d41ecbb75b P 0e4a8bb2f91540cd9e2c15bb3408f43a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:20:06.566545 11977 log.cc:1196] T 01dd37bf30b64aad9061b3d41ecbb75b P 0e4a8bb2f91540cd9e2c15bb3408f43a: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/wal/wals/01dd37bf30b64aad9061b3d41ecbb75b
I20260929 20:20:06.567008 11977 ts_tablet_manager.cc:1939] T 01dd37bf30b64aad9061b3d41ecbb75b P 0e4a8bb2f91540cd9e2c15bb3408f43a: Deleting consensus metadata
I20260929 20:20:06.567461 11975 ts_tablet_manager.cc:1918] T 01dd37bf30b64aad9061b3d41ecbb75b P ccce56dac1a84ed391bdc11b7dd6a1d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:20:06.567687 11975 log.cc:1196] T 01dd37bf30b64aad9061b3d41ecbb75b P ccce56dac1a84ed391bdc11b7dd6a1d2: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/wal/wals/01dd37bf30b64aad9061b3d41ecbb75b
I20260929 20:20:06.568284 11975 ts_tablet_manager.cc:1939] T 01dd37bf30b64aad9061b3d41ecbb75b P ccce56dac1a84ed391bdc11b7dd6a1d2: Deleting consensus metadata
I20260929 20:20:06.569074 11461 catalog_manager.cc:4841] TS 0e4a8bb2f91540cd9e2c15bb3408f43a (127.27.185.66:38641): tablet 01dd37bf30b64aad9061b3d41ecbb75b (table test-table [id=f77feeaf98e246b6bdfde0db0c197438]) successfully deleted
I20260929 20:20:06.570267 11462 catalog_manager.cc:4841] TS ccce56dac1a84ed391bdc11b7dd6a1d2 (127.27.185.67:45957): tablet 01dd37bf30b64aad9061b3d41ecbb75b (table test-table [id=f77feeaf98e246b6bdfde0db0c197438]) successfully deleted
I20260929 20:20:06.571188 11946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:06.546271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59930 (local address 127.27.185.65:42891)
0929 20:20:06.546470 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:06.546487 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:06.548018 (+ 1531us) server_negotiation.cc:407] Connection header received
0929 20:20:06.548356 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:06.548364 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:06.548476 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:06.548566 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:20:06.551262 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.552937 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.555742 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.556651 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.556743 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:06.562558 (+ 5815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:20:06.562572 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:20:06.562589 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:20:06.562638 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:20:06.566594 (+ 3956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:06.567732 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:06.567744 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:06.568198 (+ 454us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:06.568560 (+ 362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:06.569156 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:06.569161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:06.569164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:06.569677 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:20:06.569983 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:06.570568 (+ 585us) server_negotiation.cc:299] Negotiation successful
0929 20:20:06.570832 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:20:06.571993 11659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:20:06.671468 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 11536
I20260929 20:20:06.693958 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 11670
Sep 29 20:20:06 dist-test-slave-4cx3 krb5kdc[11430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.66@KRBTEST.COM
I20260929 20:20:06.709372 11949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:06.694688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56704 (local address 127.27.185.66:38641)
0929 20:20:06.694847 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:06.694852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:06.694954 (+ 102us) server_negotiation.cc:407] Connection header received
0929 20:20:06.695112 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:06.695115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:06.695177 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:06.695259 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:20:06.696576 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.697305 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.698636 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.699134 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.699297 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:06.703723 (+ 4426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:20:06.703736 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:20:06.703750 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:20:06.703787 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:20:06.706443 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:06.707147 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:06.707154 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:06.707156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:06.707285 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:06.707757 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:06.707764 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:06.707767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:06.708164 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:20:06.708473 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:06.708798 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 20:20:06.709013 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:20:06.709996 11793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:20:06.800012 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 11670
I20260929 20:20:06.817730 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 11804
Sep 29 20:20:06 dist-test-slave-4cx3 krb5kdc[11430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.67@KRBTEST.COM
I20260929 20:20:06.833591 11950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:06.818573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42362 (local address 127.27.185.67:45957)
0929 20:20:06.818762 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:06.818768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:06.818833 (+ 65us) server_negotiation.cc:407] Connection header received
0929 20:20:06.819070 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:06.819080 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:06.819156 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:06.819269 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:20:06.820624 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.821347 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.822741 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:06.823142 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:06.823280 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:06.827871 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:20:06.827888 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:20:06.827904 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:20:06.827938 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:20:06.830521 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:06.831203 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:06.831211 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:06.831214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:06.831353 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:06.831985 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:06.831992 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:06.831997 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:06.832382 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:20:06.832689 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:06.833010 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 20:20:06.833237 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:20:06.834205 11927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:20:06.924580 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 11804
I20260929 20:20:06.941649 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 11443
I20260929 20:20:06.942920 11505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:20:07.053293 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 11443
I20260929 20:20:27.075881 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:41609
--webserver_interface=127.27.185.126
--webserver_port=43017
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:43943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:41609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:20:27.414181 11983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:20:27.414563 11983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:20:27.414728 11983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:20:27.421939 11983 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 20:20:27.422086 11983 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 20:20:27.422161 11983 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 20:20:27.422259 11983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:20:27.422345 11983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:20:27.422444 11983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:20:27.422536 11983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:20:27.422614 11983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:20:27.432901 11983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:43943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:41609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:41609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=43017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:20:27.434238 11983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:20:27.435645 11983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:20:27.458768 11989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:20:27.458771 11983 server_base.cc:1034] running on GCE node
W20260929 20:20:27.458817 11992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:20:27.458726 11990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:20:27.460224 11983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:20:27.462188 11983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:20:27.463471 11983 hybrid_clock.cc:648] HybridClock initialized: now 1790713227463337 us; error 68 us; skew 500 ppm
I20260929 20:20:27.463905 11983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:20:27 dist-test-slave-4cx3 krb5kdc[11430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713227, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:20:27.467773 11983 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:20:27.469481 11983 webserver.cc:460] Webserver started at http://127.27.185.126:43017/ using document root <none> and password file <none>
I20260929 20:20:27.470305 11983 fs_manager.cc:359] Metadata directory not provided
I20260929 20:20:27.470413 11983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:20:27.476539 11983 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 20:20:27.479692 11999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:20:27.481104 11983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:20:27.481284 11983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "5562e2eff8df42f69e44202013a0c134"
format_stamp: "Formatted at 2026-09-29 20:20:04 on dist-test-slave-4cx3"
server_key: "581fafcbb6b9fb55652cf1fa48dbd301"
server_key_iv: "ea41d01177b62999a128ee64f88d6725"
server_key_version: "encryptionkey@0"
I20260929 20:20:27.482380 11983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:20:27.496693 11983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:20:27.501525 11983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:20:27.501855 11983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:20:27.525235 11983 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:41609
I20260929 20:20:27.525276 12050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:41609 every 8 connection(s)
I20260929 20:20:27.529510 11983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:20:27.531459 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 11983
I20260929 20:20:27.532510 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:42891
--local_ip_for_outbound_sockets=127.27.185.65
--tserver_master_addrs=127.27.185.126:41609
--webserver_port=39177
--webserver_interface=127.27.185.65
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:43943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:20:27.540571 12051 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:20:27.545728 12051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134: Bootstrap starting.
I20260929 20:20:27.576661 12051 log.cc:824] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134: Log is configured to *not* fsync() on all Append() calls
I20260929 20:20:27.593051 12051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:20:27.593765 12051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134: Bootstrap complete.
I20260929 20:20:27.608773 12051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5562e2eff8df42f69e44202013a0c134" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41609 } }
I20260929 20:20:27.609757 12051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5562e2eff8df42f69e44202013a0c134, State: Initialized, Role: FOLLOWER
I20260929 20:20:27.610451 12051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5562e2eff8df42f69e44202013a0c134" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41609 } }
I20260929 20:20:27.610711 12051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:20:27.610852 12051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:20:27.611037 12051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [term 1 FOLLOWER]: Advancing to term 2
I20260929 20:20:27.615183 12051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5562e2eff8df42f69e44202013a0c134" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41609 } }
I20260929 20:20:27.615805 12051 leader_election.cc:302] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5562e2eff8df42f69e44202013a0c134; no voters:
I20260929 20:20:27.616436 12051 leader_election.cc:288] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 20:20:27.616654 12055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 20:20:27.617967 12055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [term 2 LEADER]: Becoming Leader. State: Replica: 5562e2eff8df42f69e44202013a0c134, State: Running, Role: LEADER
I20260929 20:20:27.618600 12055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5562e2eff8df42f69e44202013a0c134" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41609 } }
I20260929 20:20:27.618866 12051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:20:27.621683 12057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5562e2eff8df42f69e44202013a0c134. Latest consensus state: current_term: 2 leader_uuid: "5562e2eff8df42f69e44202013a0c134" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5562e2eff8df42f69e44202013a0c134" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41609 } } }
I20260929 20:20:27.622149 12057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [sys.catalog]: This master's current role is: LEADER
I20260929 20:20:27.622803 12056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5562e2eff8df42f69e44202013a0c134" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5562e2eff8df42f69e44202013a0c134" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41609 } } }
I20260929 20:20:27.623188 12056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [sys.catalog]: This master's current role is: LEADER
I20260929 20:20:27.632979 12062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:20:27.641769 12062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:20:27.643213 12062 catalog_manager.cc:1259] Loaded cluster ID: 890e623071744b2ea5e2b009ddb3566f
I20260929 20:20:27.643321 12062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:20:27.647612 12062 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:20:27.649695 12062 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134: Loaded TSK: 0
I20260929 20:20:27.678238 12062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134: Generated new TSK 1
I20260929 20:20:27.678699 12062 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 20:20:27.937840 12053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:20:27.938146 12053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:20:27.938292 12053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:20:27.946197 12053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:20:27.946310 12053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:20:27.946481 12053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:20:27.956727 12053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:43943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:42891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=39177
--tserver_master_addrs=127.27.185.126:41609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:20:27.958135 12053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:20:27.959638 12053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:20:27.974133 12079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:20:27.974483 12081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:20:27.975122 12053 server_base.cc:1034] running on GCE node
W20260929 20:20:27.975955 12078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:20:27.976788 12053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:20:27.977928 12053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:20:27.979168 12053 hybrid_clock.cc:648] HybridClock initialized: now 1790713227979121 us; error 38 us; skew 500 ppm
I20260929 20:20:27.979617 12053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:20:27 dist-test-slave-4cx3 krb5kdc[11430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713227, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:20:27.983453 12053 init.cc:377] Logged in from keytab as kudu/127.27.185.65@KRBTEST.COM (short username kudu)
I20260929 20:20:27.985517 12053 webserver.cc:460] Webserver started at http://127.27.185.65:39177/ using document root <none> and password file <none>
I20260929 20:20:27.986590 12053 fs_manager.cc:359] Metadata directory not provided
I20260929 20:20:27.986763 12053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:20:27.995340 12053 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 20:20:27.999550 12088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:20:28.001569 12053 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 20:20:28.001781 12053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "1eaa1e9d171d400e913daf376fa7943f"
format_stamp: "Formatted at 2026-09-29 20:20:05 on dist-test-slave-4cx3"
server_key: "3f1f32d5303ba34e16b76822fd6592af"
server_key_iv: "e2f9e954fdfc11fd81a9650954cbc072"
server_key_version: "encryptionkey@0"
I20260929 20:20:28.003435 12053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:20:28.040807 12053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:20:28.047040 12053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:20:28.047366 12053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:20:28.048671 12053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:20:28.050540 12053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:20:28.050668 12053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:20:28.050784 12053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:20:28.050885 12053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:20:28.091275 12053 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:42891
I20260929 20:20:28.091336 12200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:42891 every 8 connection(s)
I20260929 20:20:28.093029 12053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:20:28.096222 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 12053
I20260929 20:20:28.097074 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:38641
--local_ip_for_outbound_sockets=127.27.185.66
--tserver_master_addrs=127.27.185.126:41609
--webserver_port=45655
--webserver_interface=127.27.185.66
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:43943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:20:28 dist-test-slave-4cx3 krb5kdc[11430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713227, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:20:28.122104 12204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:28.096525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.65:40959)
0929 20:20:28.097510 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:28.097568 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 20:20:28.099566 (+ 1998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:20:28.101229 (+ 1663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:20:28.101237 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:20:28.101832 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:20:28.102619 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:28.102644 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:28.105187 (+ 2543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:28.105194 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:28.106925 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:28.106939 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:28.108567 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:28.108572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:28.108782 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:28.109660 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:20:28.109685 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:20:28.112172 (+ 2487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:20:28.117107 (+ 4935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:20:28.117115 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:20:28.117127 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:20:28.117713 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:20:28.118288 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:20:28.118291 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:20:28.118293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:20:28.118501 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:20:28.119730 (+ 1229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:20:28.119737 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:20:28.120158 (+ 421us) client_negotiation.cc:769] Sending connection context
0929 20:20:28.120522 (+ 364us) client_negotiation.cc:240] Negotiation successful
0929 20:20:28.120973 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":418,"threads_started":1}
I20260929 20:20:28.122743 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:28.096229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:40959 (local address 127.27.185.126:41609)
0929 20:20:28.096915 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:28.096935 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:28.097729 (+ 794us) server_negotiation.cc:407] Connection header received
0929 20:20:28.100167 (+ 2438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:28.100190 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:28.100648 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:28.101086 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:20:28.102897 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:28.104980 (+ 2083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:28.107096 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:28.108323 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:28.108540 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:28.112510 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:20:28.112523 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:20:28.112547 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:20:28.112603 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:20:28.116900 (+ 4297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:28.117876 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:28.117890 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:28.117900 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:28.118162 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:28.118621 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:28.118628 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:28.118633 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:28.119535 (+ 902us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:20:28.119769 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:28.120791 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0929 20:20:28.121326 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":280,"threads_started":1}
I20260929 20:20:28.125782 12201 heartbeater.cc:347] Connected to a master server at 127.27.185.126:41609
I20260929 20:20:28.126133 12201 heartbeater.cc:464] Registering TS with master...
I20260929 20:20:28.126991 12201 heartbeater.cc:511] Master 127.27.185.126:41609 requested a full tablet report, sending...
I20260929 20:20:28.129372 12016 ts_manager.cc:194] Registered new tserver with Master: 1eaa1e9d171d400e913daf376fa7943f (127.27.185.65:42891)
I20260929 20:20:28.130980 12016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.65@KRBTEST.COM'} at 127.27.185.65:40959
W20260929 20:20:28.440500 12205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:20:28.440824 12205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:20:28.440985 12205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:20:28.448523 12205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:20:28.448645 12205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:20:28.448832 12205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:20:28.458832 12205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:43943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:38641
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=45655
--tserver_master_addrs=127.27.185.126:41609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:20:28.460191 12205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:20:28.461561 12205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:20:28.476192 12210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:20:28.476260 12211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:20:28.476235 12213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:20:28.477541 12205 server_base.cc:1034] running on GCE node
I20260929 20:20:28.478180 12205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:20:28.479157 12205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:20:28.480387 12205 hybrid_clock.cc:648] HybridClock initialized: now 1790713228480273 us; error 108 us; skew 500 ppm
I20260929 20:20:28.480722 12205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:20:28 dist-test-slave-4cx3 krb5kdc[11430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713228, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:20:28.490085 12205 init.cc:377] Logged in from keytab as kudu/127.27.185.66@KRBTEST.COM (short username kudu)
I20260929 20:20:28.491820 12205 webserver.cc:460] Webserver started at http://127.27.185.66:45655/ using document root <none> and password file <none>
I20260929 20:20:28.492569 12205 fs_manager.cc:359] Metadata directory not provided
I20260929 20:20:28.492733 12205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:20:28.498261 12205 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.003s
I20260929 20:20:28.501148 12220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:20:28.502681 12205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:20:28.502880 12205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a"
format_stamp: "Formatted at 2026-09-29 20:20:05 on dist-test-slave-4cx3"
server_key: "67af568a608460a635d7b4363445ee6a"
server_key_iv: "d3666038e879091cc8912c746c0d68c2"
server_key_version: "encryptionkey@0"
I20260929 20:20:28.503966 12205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:20:28.521940 12205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:20:28.526525 12205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:20:28.526834 12205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:20:28.527894 12205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:20:28.529390 12205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:20:28.529508 12205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:20:28.529637 12205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:20:28.529729 12205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:20:28.564666 12205 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:38641
I20260929 20:20:28.564733 12332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:38641 every 8 connection(s)
I20260929 20:20:28.566339 12205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:20:28.576017 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 12205
I20260929 20:20:28.576690 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:45957
--local_ip_for_outbound_sockets=127.27.185.67
--tserver_master_addrs=127.27.185.126:41609
--webserver_port=35807
--webserver_interface=127.27.185.67
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:43943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:20:28 dist-test-slave-4cx3 krb5kdc[11430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713228, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:20:28.589248 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:28.569177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:34795 (local address 127.27.185.126:41609)
0929 20:20:28.569414 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:28.569419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:28.570690 (+ 1271us) server_negotiation.cc:407] Connection header received
0929 20:20:28.572676 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:28.572681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:28.572780 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:28.572892 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:20:28.574612 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:28.575337 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:28.577232 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:28.578063 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:28.578220 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:28.582157 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:20:28.582166 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:20:28.582174 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:20:28.582227 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:20:28.585448 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:28.586532 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:28.586538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:28.586542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:28.586690 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:28.587281 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:28.587286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:28.587290 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:28.587689 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:20:28.587879 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:28.588656 (+ 777us) server_negotiation.cc:299] Negotiation successful
0929 20:20:28.588900 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:20:28.590361 12335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:28.569740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.66:34795)
0929 20:20:28.570506 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:28.570549 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 20:20:28.572367 (+ 1818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:20:28.573168 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:20:28.573177 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:20:28.573796 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:20:28.574421 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:28.574440 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:28.575575 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:28.575580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:28.577066 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:28.577081 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:28.578321 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:28.578326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:28.578576 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:28.579539 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:20:28.579563 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:20:28.581856 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:20:28.585639 (+ 3783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:20:28.585652 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:20:28.585674 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:20:28.586399 (+ 725us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:20:28.586921 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:20:28.586925 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:20:28.586927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:20:28.587149 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:20:28.587842 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:20:28.587851 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:20:28.588264 (+ 413us) client_negotiation.cc:769] Sending connection context
0929 20:20:28.588549 (+ 285us) client_negotiation.cc:240] Negotiation successful
0929 20:20:28.589146 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":274,"threads_started":1}
I20260929 20:20:28.592059 12333 heartbeater.cc:347] Connected to a master server at 127.27.185.126:41609
I20260929 20:20:28.592425 12333 heartbeater.cc:464] Registering TS with master...
I20260929 20:20:28.593374 12333 heartbeater.cc:511] Master 127.27.185.126:41609 requested a full tablet report, sending...
I20260929 20:20:28.595081 12016 ts_manager.cc:194] Registered new tserver with Master: 0e4a8bb2f91540cd9e2c15bb3408f43a (127.27.185.66:38641)
I20260929 20:20:28.596086 12016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.66@KRBTEST.COM'} at 127.27.185.66:34795
W20260929 20:20:28.919200 12336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:20:28.919513 12336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:20:28.919706 12336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:20:28.927194 12336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:20:28.927337 12336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:20:28.927547 12336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:20:28.938206 12336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:43943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:45957
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=35807
--tserver_master_addrs=127.27.185.126:41609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:20:28.939595 12336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:20:28.940920 12336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:20:28.954414 12341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:20:28.954442 12342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:20:28.954492 12344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:20:28.955461 12336 server_base.cc:1034] running on GCE node
I20260929 20:20:28.956065 12336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:20:28.956961 12336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:20:28.958170 12336 hybrid_clock.cc:648] HybridClock initialized: now 1790713228958126 us; error 29 us; skew 500 ppm
I20260929 20:20:28.958453 12336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:20:28 dist-test-slave-4cx3 krb5kdc[11430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713228, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:20:28.962054 12336 init.cc:377] Logged in from keytab as kudu/127.27.185.67@KRBTEST.COM (short username kudu)
I20260929 20:20:28.963632 12336 webserver.cc:460] Webserver started at http://127.27.185.67:35807/ using document root <none> and password file <none>
I20260929 20:20:28.964357 12336 fs_manager.cc:359] Metadata directory not provided
I20260929 20:20:28.964521 12336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:20:28.970364 12336 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 20:20:28.973321 12351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:20:28.974790 12336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:20:28.974977 12336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2"
format_stamp: "Formatted at 2026-09-29 20:20:06 on dist-test-slave-4cx3"
server_key: "5aebf81f95b9b81b6ed0a11724b8569a"
server_key_iv: "fcb66ae6f8487cbc9808933eea0c13a0"
server_key_version: "encryptionkey@0"
I20260929 20:20:28.976090 12336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:20:28.992185 12336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:20:28.996304 12336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:20:28.996598 12336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:20:28.997618 12336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:20:28.999032 12336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:20:28.999161 12336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:20:28.999289 12336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:20:28.999377 12336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:20:29.034644 12336 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:45957
I20260929 20:20:29.034758 12463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:45957 every 8 connection(s)
I20260929 20:20:29.036208 12336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:20:29.044606 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 12336
Sep 29 20:20:29 dist-test-slave-4cx3 krb5kdc[11430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713228, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:20:29.059759 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:29.038971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:37761 (local address 127.27.185.126:41609)
0929 20:20:29.039158 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:29.039162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:29.040358 (+ 1196us) server_negotiation.cc:407] Connection header received
0929 20:20:29.042200 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:29.042206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:29.042304 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:29.042416 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:20:29.044019 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.044810 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.046844 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.047680 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.047855 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:29.052006 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:20:29.052011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:20:29.052018 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:20:29.052053 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:20:29.055724 (+ 3671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:29.056740 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:29.056747 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:29.056752 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:29.056933 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:29.057495 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:29.057499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:29.057506 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:29.057957 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:20:29.058085 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:29.059052 (+ 967us) server_negotiation.cc:299] Negotiation successful
0929 20:20:29.059311 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:20:29.061053 12466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:29.039517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.67:37761)
0929 20:20:29.040200 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:29.040239 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 20:20:29.041922 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:20:29.042674 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:20:29.042682 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:20:29.043199 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:20:29.043848 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:29.043865 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.044976 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.044980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:29.046672 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:29.046690 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.047917 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.047923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:29.048133 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:29.048894 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:20:29.048931 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:20:29.051756 (+ 2825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:20:29.055887 (+ 4131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:20:29.055896 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:20:29.055915 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:20:29.056586 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:20:29.057074 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:20:29.057081 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:20:29.057087 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:20:29.057342 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:20:29.058100 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:20:29.058113 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:20:29.058603 (+ 490us) client_negotiation.cc:769] Sending connection context
0929 20:20:29.058938 (+ 335us) client_negotiation.cc:240] Negotiation successful
0929 20:20:29.059581 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":258,"threads_started":1}
I20260929 20:20:29.062668 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:29.045546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52590 (local address 127.27.185.126:41609)
0929 20:20:29.046055 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:29.046061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:29.046378 (+ 317us) server_negotiation.cc:407] Connection header received
0929 20:20:29.046524 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:29.046529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:29.046605 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:29.046721 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:20:29.048852 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.049987 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.051779 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.052304 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.052397 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:29.055939 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:20:29.055945 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:20:29.055949 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:20:29.055987 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:20:29.059430 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:29.060297 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:29.060302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:29.060305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:29.060436 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:29.060807 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:29.060811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:29.060814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:29.061168 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:20:29.061419 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:29.062064 (+ 645us) server_negotiation.cc:299] Negotiation successful
0929 20:20:29.062304 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":230,"threads_started":1}
I20260929 20:20:29.062891 12464 heartbeater.cc:347] Connected to a master server at 127.27.185.126:41609
I20260929 20:20:29.063242 12464 heartbeater.cc:464] Registering TS with master...
I20260929 20:20:29.064055 12464 heartbeater.cc:511] Master 127.27.185.126:41609 requested a full tablet report, sending...
I20260929 20:20:29.065547 12016 ts_manager.cc:194] Registered new tserver with Master: ccce56dac1a84ed391bdc11b7dd6a1d2 (127.27.185.67:45957)
I20260929 20:20:29.066416 12016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.67@KRBTEST.COM'} at 127.27.185.67:37761
I20260929 20:20:29.078115 28389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:20:29.085702 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:29.079153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52596 (local address 127.27.185.126:41609)
0929 20:20:29.079494 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:29.079505 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:29.079994 (+ 489us) server_negotiation.cc:407] Connection header received
0929 20:20:29.080192 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:29.080196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:29.080347 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:29.080549 (+ 202us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:20:29.081033 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.081809 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.083473 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.083951 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.084103 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:29.084663 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:20:29.084900 (+ 237us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 20:20:29.085322 (+ 422us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52596: token expired
Metrics: {"server-negotiator.queue_time_us":131}
W20260929 20:20:29.085851 12467 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52596: token expired
W20260929 20:20:29.086309 11937 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.27.185.126:41609)
I20260929 20:20:29.101893 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:29.086991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52612 (local address 127.27.185.126:41609)
0929 20:20:29.087170 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:29.087174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:29.087194 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:20:29.087322 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:29.087325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:29.087430 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:29.087517 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:20:29.089514 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.090251 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.092579 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.093001 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.093104 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:29.096143 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:20:29.096148 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:20:29.096151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:20:29.096191 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:20:29.098897 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:29.099815 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:29.099819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:29.099821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:29.099975 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:29.100368 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:29.100373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:29.100375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:29.100721 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:20:29.100935 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:29.101272 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 20:20:29.101500 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:20:29.105696 12016 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:20:29.114152 12016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:20:29.135015 12201 heartbeater.cc:503] Master 127.27.185.126:41609 was elected leader, sending a full tablet report...
I20260929 20:20:29.161597 12476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:29.127346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:42891 (local address 127.0.0.1:56686)
0929 20:20:29.152125 (+ 24779us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:29.152147 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:20:29.152249 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:20:29.154028 (+ 1779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:20:29.154038 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:20:29.154075 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:20:29.154526 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:29.154540 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.156727 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.156732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:29.158516 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:29.158530 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.160523 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.160529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:29.160804 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:29.160870 (+ 66us) client_negotiation.cc:769] Sending connection context
0929 20:20:29.160985 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 20:20:29.161199 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24653,"thread_start_us":24812,"threads_started":1}
I20260929 20:20:29.162173 12477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:29.127566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56686 (local address 127.27.185.65:42891)
0929 20:20:29.152772 (+ 25206us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:29.152781 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:29.152812 (+ 31us) server_negotiation.cc:407] Connection header received
0929 20:20:29.152959 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:29.152967 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:29.153238 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:29.153460 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:20:29.154899 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.156561 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.158683 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.160235 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.161044 (+ 809us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:29.161286 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:29.161366 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 20:20:29.161529 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25038,"thread_start_us":26720,"threads_started":1}
I20260929 20:20:29.164142 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:29.127319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45440 (local address 127.27.185.66:38641)
0929 20:20:29.154279 (+ 26960us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:29.154288 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:29.154316 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:20:29.154428 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:29.154436 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:29.154712 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:29.155552 (+ 840us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:20:29.157003 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.159008 (+ 2005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.161653 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.162976 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.163145 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:29.163341 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:29.163661 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 20:20:29.163826 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26792,"thread_start_us":26667,"threads_started":1}
I20260929 20:20:29.164142 12478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:29.127298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:38641 (local address 127.0.0.1:45440)
0929 20:20:29.151601 (+ 24303us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:29.151662 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 20:20:29.151952 (+ 290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:20:29.154941 (+ 2989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:20:29.154947 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:20:29.155893 (+ 946us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:20:29.156381 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:29.156399 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.159181 (+ 2782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.159186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:29.161428 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:29.161443 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.163226 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.163233 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:29.163454 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:29.163482 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 20:20:29.163587 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 20:20:29.163729 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23983,"mutex_wait_us":1,"thread_start_us":31502,"threads_started":1}
I20260929 20:20:29.166554 12136 tablet_service.cc:1467] Processing CreateTablet for tablet 4b818ba1c5b34941ae41c09fd91eae9a (DEFAULT_TABLE table=test-table [id=1ef7a579050645709c34d7368ecc9484]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:20:29.168007 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:29.129756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:45957 (local address 127.0.0.1:45660)
0929 20:20:29.159096 (+ 29340us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:29.159116 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:20:29.159265 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:20:29.160222 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:20:29.160226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:20:29.160252 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:20:29.160651 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:29.160667 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.162602 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.162607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:29.164965 (+ 2358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:29.164980 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.167125 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.167131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:29.167307 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:29.167337 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 20:20:29.167514 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 20:20:29.167649 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29200,"thread_start_us":23923,"threads_started":1}
I20260929 20:20:29.168138 12479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:29.130109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45660 (local address 127.27.185.67:45957)
0929 20:20:29.151435 (+ 21326us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:29.151446 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:29.159274 (+ 7828us) server_negotiation.cc:407] Connection header received
0929 20:20:29.159424 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:29.159434 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:29.159697 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:29.159896 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:20:29.160866 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.162459 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.165164 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.166602 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.166710 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:29.166927 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:29.167599 (+ 672us) server_negotiation.cc:299] Negotiation successful
0929 20:20:29.167782 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21078,"thread_start_us":21177,"threads_started":1}
I20260929 20:20:29.169085 12136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b818ba1c5b34941ae41c09fd91eae9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:20:29.170243 12268 tablet_service.cc:1467] Processing CreateTablet for tablet 4b818ba1c5b34941ae41c09fd91eae9a (DEFAULT_TABLE table=test-table [id=1ef7a579050645709c34d7368ecc9484]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:20:29.171736 12268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b818ba1c5b34941ae41c09fd91eae9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:20:29.171826 12399 tablet_service.cc:1467] Processing CreateTablet for tablet 4b818ba1c5b34941ae41c09fd91eae9a (DEFAULT_TABLE table=test-table [id=1ef7a579050645709c34d7368ecc9484]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:20:29.173221 12399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b818ba1c5b34941ae41c09fd91eae9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:20:29.182317 12480 tablet_bootstrap.cc:492] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f: Bootstrap starting.
I20260929 20:20:29.185791 12480 tablet_bootstrap.cc:654] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f: Neither blocks nor log segments found. Creating new log.
I20260929 20:20:29.187537 12480 log.cc:824] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:20:29.187628 12481 tablet_bootstrap.cc:492] T 4b818ba1c5b34941ae41c09fd91eae9a P 0e4a8bb2f91540cd9e2c15bb3408f43a: Bootstrap starting.
I20260929 20:20:29.191593 12481 tablet_bootstrap.cc:654] T 4b818ba1c5b34941ae41c09fd91eae9a P 0e4a8bb2f91540cd9e2c15bb3408f43a: Neither blocks nor log segments found. Creating new log.
I20260929 20:20:29.191591 12482 tablet_bootstrap.cc:492] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2: Bootstrap starting.
I20260929 20:20:29.191774 12480 tablet_bootstrap.cc:492] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f: No bootstrap required, opened a new log
I20260929 20:20:29.192059 12480 ts_tablet_manager.cc:1397] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260929 20:20:29.192919 12481 log.cc:824] T 4b818ba1c5b34941ae41c09fd91eae9a P 0e4a8bb2f91540cd9e2c15bb3408f43a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:20:29.194844 12482 tablet_bootstrap.cc:654] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2: Neither blocks nor log segments found. Creating new log.
I20260929 20:20:29.196132 12482 log.cc:824] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2: Log is configured to *not* fsync() on all Append() calls
I20260929 20:20:29.196226 12481 tablet_bootstrap.cc:492] T 4b818ba1c5b34941ae41c09fd91eae9a P 0e4a8bb2f91540cd9e2c15bb3408f43a: No bootstrap required, opened a new log
I20260929 20:20:29.196482 12481 ts_tablet_manager.cc:1397] T 4b818ba1c5b34941ae41c09fd91eae9a P 0e4a8bb2f91540cd9e2c15bb3408f43a: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260929 20:20:29.199270 12482 tablet_bootstrap.cc:492] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2: No bootstrap required, opened a new log
I20260929 20:20:29.199575 12482 ts_tablet_manager.cc:1397] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 20:20:29.200650 12480 raft_consensus.cc:348] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38641 } } peers { permanent_uuid: "1eaa1e9d171d400e913daf376fa7943f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42891 } } peers { permanent_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45957 } }
I20260929 20:20:29.201040 12480 raft_consensus.cc:374] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:20:29.201148 12480 raft_consensus.cc:729] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eaa1e9d171d400e913daf376fa7943f, State: Initialized, Role: FOLLOWER
I20260929 20:20:29.201812 12480 consensus_queue.cc:260] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38641 } } peers { permanent_uuid: "1eaa1e9d171d400e913daf376fa7943f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42891 } } peers { permanent_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45957 } }
I20260929 20:20:29.203706 12480 ts_tablet_manager.cc:1428] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260929 20:20:29.207083 12481 raft_consensus.cc:348] T 4b818ba1c5b34941ae41c09fd91eae9a P 0e4a8bb2f91540cd9e2c15bb3408f43a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38641 } } peers { permanent_uuid: "1eaa1e9d171d400e913daf376fa7943f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42891 } } peers { permanent_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45957 } }
I20260929 20:20:29.207583 12481 raft_consensus.cc:374] T 4b818ba1c5b34941ae41c09fd91eae9a P 0e4a8bb2f91540cd9e2c15bb3408f43a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:20:29.207690 12481 raft_consensus.cc:729] T 4b818ba1c5b34941ae41c09fd91eae9a P 0e4a8bb2f91540cd9e2c15bb3408f43a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4a8bb2f91540cd9e2c15bb3408f43a, State: Initialized, Role: FOLLOWER
I20260929 20:20:29.208433 12481 consensus_queue.cc:260] T 4b818ba1c5b34941ae41c09fd91eae9a P 0e4a8bb2f91540cd9e2c15bb3408f43a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38641 } } peers { permanent_uuid: "1eaa1e9d171d400e913daf376fa7943f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42891 } } peers { permanent_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45957 } }
I20260929 20:20:29.209978 12333 heartbeater.cc:503] Master 127.27.185.126:41609 was elected leader, sending a full tablet report...
I20260929 20:20:29.210378 12481 ts_tablet_manager.cc:1428] T 4b818ba1c5b34941ae41c09fd91eae9a P 0e4a8bb2f91540cd9e2c15bb3408f43a: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260929 20:20:29.210140 12482 raft_consensus.cc:348] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38641 } } peers { permanent_uuid: "1eaa1e9d171d400e913daf376fa7943f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42891 } } peers { permanent_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45957 } }
I20260929 20:20:29.210601 12482 raft_consensus.cc:374] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:20:29.210714 12482 raft_consensus.cc:729] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccce56dac1a84ed391bdc11b7dd6a1d2, State: Initialized, Role: FOLLOWER
I20260929 20:20:29.211252 12482 consensus_queue.cc:260] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38641 } } peers { permanent_uuid: "1eaa1e9d171d400e913daf376fa7943f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42891 } } peers { permanent_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45957 } }
I20260929 20:20:29.212399 12464 heartbeater.cc:503] Master 127.27.185.126:41609 was elected leader, sending a full tablet report...
I20260929 20:20:29.212925 12482 ts_tablet_manager.cc:1428] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260929 20:20:29.287963 12465 tablet.cc:2369] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:20:29.318544 12334 tablet.cc:2369] T 4b818ba1c5b34941ae41c09fd91eae9a P 0e4a8bb2f91540cd9e2c15bb3408f43a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:20:29.337090 12488 raft_consensus.cc:482] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:20:29.337389 12488 raft_consensus.cc:504] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38641 } } peers { permanent_uuid: "1eaa1e9d171d400e913daf376fa7943f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42891 } } peers { permanent_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45957 } }
I20260929 20:20:29.339051 12488 leader_election.cc:288] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e4a8bb2f91540cd9e2c15bb3408f43a (127.27.185.66:38641), 1eaa1e9d171d400e913daf376fa7943f (127.27.185.65:42891)
W20260929 20:20:29.345582 12202 tablet.cc:2369] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:20:29.346560 12466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:29.339448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:38641 (local address 127.27.185.67:35919)
0929 20:20:29.339829 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:29.339850 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:20:29.340012 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:20:29.340486 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:20:29.340492 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:20:29.340520 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:20:29.340679 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:29.340687 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.341994 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.341998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:29.344169 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:29.344185 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.345735 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.345741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:29.345932 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:29.345968 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 20:20:29.346076 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 20:20:29.346222 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":49}
I20260929 20:20:29.346594 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:29.339519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:35919 (local address 127.27.185.66:38641)
0929 20:20:29.339737 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:29.339745 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:29.339996 (+ 251us) server_negotiation.cc:407] Connection header received
0929 20:20:29.340193 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:29.340200 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:29.340287 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:29.340366 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:20:29.340866 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.341861 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.344341 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.345554 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.345729 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:29.345815 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:29.346166 (+ 351us) server_negotiation.cc:299] Negotiation successful
0929 20:20:29.346303 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:20:29.347209 12477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:29.339657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:54753 (local address 127.27.185.65:42891)
0929 20:20:29.339912 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:29.339921 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:29.340404 (+ 483us) server_negotiation.cc:407] Connection header received
0929 20:20:29.340589 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:29.340593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:29.340651 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:29.340720 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:20:29.341254 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.342207 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.344555 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.345850 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.345997 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:29.346077 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:29.346498 (+ 421us) server_negotiation.cc:299] Negotiation successful
0929 20:20:29.346966 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 20:20:29.347621 12489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:29.339553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:42891 (local address 127.27.185.67:54753)
0929 20:20:29.340288 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:29.340301 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 20:20:29.340439 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:20:29.340772 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:20:29.340778 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:20:29.340804 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:20:29.341099 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:29.341107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.342349 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.342354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:29.344403 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:29.344412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.346122 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.346129 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:29.346303 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:29.346336 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:20:29.347058 (+ 722us) client_negotiation.cc:240] Negotiation successful
0929 20:20:29.347193 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":1,"thread_start_us":444,"threads_started":1}
I20260929 20:20:29.347780 12288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b818ba1c5b34941ae41c09fd91eae9a" candidate_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a" is_pre_election: true
I20260929 20:20:29.348380 12288 raft_consensus.cc:2454] T 4b818ba1c5b34941ae41c09fd91eae9a P 0e4a8bb2f91540cd9e2c15bb3408f43a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccce56dac1a84ed391bdc11b7dd6a1d2 in term 0.
I20260929 20:20:29.348789 12156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b818ba1c5b34941ae41c09fd91eae9a" candidate_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eaa1e9d171d400e913daf376fa7943f" is_pre_election: true
I20260929 20:20:29.349213 12354 leader_election.cc:302] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4a8bb2f91540cd9e2c15bb3408f43a, ccce56dac1a84ed391bdc11b7dd6a1d2; no voters:
I20260929 20:20:29.349406 12156 raft_consensus.cc:2454] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccce56dac1a84ed391bdc11b7dd6a1d2 in term 0.
I20260929 20:20:29.349647 12488 raft_consensus.cc:2781] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:20:29.349822 12488 raft_consensus.cc:482] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:20:29.349956 12488 raft_consensus.cc:3037] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:20:29.352257 12488 raft_consensus.cc:504] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38641 } } peers { permanent_uuid: "1eaa1e9d171d400e913daf376fa7943f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42891 } } peers { permanent_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45957 } }
I20260929 20:20:29.352944 12488 leader_election.cc:288] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2 [CANDIDATE]: Term 1 election: Requested vote from peers 0e4a8bb2f91540cd9e2c15bb3408f43a (127.27.185.66:38641), 1eaa1e9d171d400e913daf376fa7943f (127.27.185.65:42891)
I20260929 20:20:29.353487 12288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b818ba1c5b34941ae41c09fd91eae9a" candidate_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a"
I20260929 20:20:29.353559 12156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b818ba1c5b34941ae41c09fd91eae9a" candidate_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eaa1e9d171d400e913daf376fa7943f"
I20260929 20:20:29.353756 12156 raft_consensus.cc:3037] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:20:29.353749 12288 raft_consensus.cc:3037] T 4b818ba1c5b34941ae41c09fd91eae9a P 0e4a8bb2f91540cd9e2c15bb3408f43a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:20:29.357159 12288 raft_consensus.cc:2454] T 4b818ba1c5b34941ae41c09fd91eae9a P 0e4a8bb2f91540cd9e2c15bb3408f43a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccce56dac1a84ed391bdc11b7dd6a1d2 in term 1.
I20260929 20:20:29.357159 12156 raft_consensus.cc:2454] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccce56dac1a84ed391bdc11b7dd6a1d2 in term 1.
I20260929 20:20:29.357754 12354 leader_election.cc:302] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4a8bb2f91540cd9e2c15bb3408f43a, ccce56dac1a84ed391bdc11b7dd6a1d2; no voters:
I20260929 20:20:29.358147 12488 raft_consensus.cc:2781] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:20:29.358693 12488 raft_consensus.cc:686] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2 [term 1 LEADER]: Becoming Leader. State: Replica: ccce56dac1a84ed391bdc11b7dd6a1d2, State: Running, Role: LEADER
I20260929 20:20:29.359227 12488 consensus_queue.cc:237] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38641 } } peers { permanent_uuid: "1eaa1e9d171d400e913daf376fa7943f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42891 } } peers { permanent_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45957 } }
I20260929 20:20:29.364399 12015 catalog_manager.cc:5495] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to ccce56dac1a84ed391bdc11b7dd6a1d2 (127.27.185.67). New cstate: current_term: 1 leader_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eaa1e9d171d400e913daf376fa7943f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45957 } health_report { overall_health: HEALTHY } } }
I20260929 20:20:29.405936 12479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:29.400733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45676 (local address 127.27.185.67:45957)
0929 20:20:29.400983 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:29.400989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:29.401015 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:20:29.401216 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:29.401220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:29.401296 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:29.401427 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:20:29.401764 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.402514 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.403822 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.404225 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.404323 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:29.404813 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:20:29.405343 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:20:29.405471 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:29.405553 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 20:20:29.405621 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 20:20:29.414273 12156 raft_consensus.cc:1261] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [term 1 FOLLOWER]: Refusing update from remote peer ccce56dac1a84ed391bdc11b7dd6a1d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:20:29.414271 12288 raft_consensus.cc:1261] T 4b818ba1c5b34941ae41c09fd91eae9a P 0e4a8bb2f91540cd9e2c15bb3408f43a [term 1 FOLLOWER]: Refusing update from remote peer ccce56dac1a84ed391bdc11b7dd6a1d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:20:29.415465 12490 consensus_queue.cc:1035] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:20:29.415831 12488 consensus_queue.cc:1035] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eaa1e9d171d400e913daf376fa7943f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:20:29.432852 12493 mvcc.cc:205] Tried to move back new op lower bound from 7334761387671150592 to 7334761387459682304. Current Snapshot: MvccSnapshot[applied={T|T < 7334761387671150592}]
I20260929 20:20:29.435495 12495 mvcc.cc:205] Tried to move back new op lower bound from 7334761387671150592 to 7334761387459682304. Current Snapshot: MvccSnapshot[applied={T|T < 7334761387671150592}]
I20260929 20:20:29.446748 12015 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:20:29.447065 12015 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:20:29.450769 12015 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134: Sending DeleteTablet for 3 replicas of tablet 4b818ba1c5b34941ae41c09fd91eae9a
I20260929 20:20:29.452082 12136 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b818ba1c5b34941ae41c09fd91eae9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:20:29 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:56686
I20260929 20:20:29.452073 12399 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b818ba1c5b34941ae41c09fd91eae9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:20:29 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:45660
I20260929 20:20:29.452131 12268 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b818ba1c5b34941ae41c09fd91eae9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:20:29 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:45440
I20260929 20:20:29.456015 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 12053
I20260929 20:20:29.466221 12503 tablet_replica.cc:330] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2: stopping tablet replica
I20260929 20:20:29.466874 12503 raft_consensus.cc:2229] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:20:29.467881 12503 raft_consensus.cc:2258] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:20:29.471627 12503 ts_tablet_manager.cc:1905] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:20:29.472285 12504 tablet_replica.cc:330] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f: stopping tablet replica
I20260929 20:20:29.473089 12504 raft_consensus.cc:2229] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:20:29.473657 12504 raft_consensus.cc:2258] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:20:29.477152 12504 ts_tablet_manager.cc:1905] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:20:29.478869 12477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:29.457263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56694 (local address 127.27.185.65:42891)
0929 20:20:29.457438 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:29.457445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:29.457463 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:20:29.457527 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:29.457532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:29.457613 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:29.457716 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:20:29.459710 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.461013 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.463343 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.464087 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.464217 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:29.468582 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:20:29.468605 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:20:29.468626 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:20:29.468674 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:20:29.474023 (+ 5349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:29.475265 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:29.475280 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:29.475286 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:29.475534 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:29.476036 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:29.476041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:29.476044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:29.476506 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:20:29.476800 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:29.478200 (+ 1400us) server_negotiation.cc:299] Negotiation successful
0929 20:20:29.478463 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:20:29.479928 12196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:20:29.480485 12505 tablet_replica.cc:330] T 4b818ba1c5b34941ae41c09fd91eae9a P 0e4a8bb2f91540cd9e2c15bb3408f43a: stopping tablet replica
I20260929 20:20:29.481122 12505 raft_consensus.cc:2229] T 4b818ba1c5b34941ae41c09fd91eae9a P 0e4a8bb2f91540cd9e2c15bb3408f43a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:20:29.481724 12505 raft_consensus.cc:2258] T 4b818ba1c5b34941ae41c09fd91eae9a P 0e4a8bb2f91540cd9e2c15bb3408f43a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:20:29.481794 12503 ts_tablet_manager.cc:1918] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:20:29.481983 12503 log.cc:1196] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/wal/wals/4b818ba1c5b34941ae41c09fd91eae9a
I20260929 20:20:29.482565 12503 ts_tablet_manager.cc:1939] T 4b818ba1c5b34941ae41c09fd91eae9a P ccce56dac1a84ed391bdc11b7dd6a1d2: Deleting consensus metadata
I20260929 20:20:29.484265 12505 ts_tablet_manager.cc:1905] T 4b818ba1c5b34941ae41c09fd91eae9a P 0e4a8bb2f91540cd9e2c15bb3408f43a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:20:29.484501 12003 catalog_manager.cc:4841] TS ccce56dac1a84ed391bdc11b7dd6a1d2 (127.27.185.67:45957): tablet 4b818ba1c5b34941ae41c09fd91eae9a (table test-table [id=1ef7a579050645709c34d7368ecc9484]) successfully deleted
I20260929 20:20:29.492372 12505 ts_tablet_manager.cc:1918] T 4b818ba1c5b34941ae41c09fd91eae9a P 0e4a8bb2f91540cd9e2c15bb3408f43a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:20:29.492558 12505 log.cc:1196] T 4b818ba1c5b34941ae41c09fd91eae9a P 0e4a8bb2f91540cd9e2c15bb3408f43a: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/wal/wals/4b818ba1c5b34941ae41c09fd91eae9a
I20260929 20:20:29.493129 12505 ts_tablet_manager.cc:1939] T 4b818ba1c5b34941ae41c09fd91eae9a P 0e4a8bb2f91540cd9e2c15bb3408f43a: Deleting consensus metadata
I20260929 20:20:29.495249 12002 catalog_manager.cc:4841] TS 0e4a8bb2f91540cd9e2c15bb3408f43a (127.27.185.66:38641): tablet 4b818ba1c5b34941ae41c09fd91eae9a (table test-table [id=1ef7a579050645709c34d7368ecc9484]) successfully deleted
I20260929 20:20:29.592916 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 12053
W20260929 20:20:29.611491 12000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.185.65:42891 (error 108)
I20260929 20:20:29.611734 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 12205
I20260929 20:20:29.614854 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:29.614355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:42891 (local address 127.0.0.1:56706)
0929 20:20:29.614548 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:29.614672 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:42891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 20:20:29.615110 12000 catalog_manager.cc:4584] TS 1eaa1e9d171d400e913daf376fa7943f (127.27.185.65:42891): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4b818ba1c5b34941ae41c09fd91eae9a: Network error: Client connection negotiation failed: client connection to 127.27.185.65:42891: connect: Connection refused (error 111)
I20260929 20:20:29.626951 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:29.612893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45446 (local address 127.27.185.66:38641)
0929 20:20:29.613071 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:29.613077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:29.613099 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:20:29.613164 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:29.613170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:29.613251 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:29.613374 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:20:29.615128 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.616144 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.617502 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.617889 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.618029 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:29.620963 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:20:29.620976 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:20:29.620991 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:20:29.621024 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:20:29.624174 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:29.624879 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:29.624889 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:29.624892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:29.625008 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:29.625387 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:29.625392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:29.625397 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:29.625778 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:20:29.626081 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:29.626399 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 20:20:29.626614 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:20:29.627704 12328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:20:29.668931 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:29.668373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:42891 (local address 127.0.0.1:56708)
0929 20:20:29.668625 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:29.668753 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:42891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 20:20:29.670212 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:29.669748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:42891 (local address 127.0.0.1:56712)
0929 20:20:29.669919 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:29.670045 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:42891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 20:20:29.718771 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 12205
I20260929 20:20:29.731305 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:29.730660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:42891 (local address 127.0.0.1:56728)
0929 20:20:29.730947 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:29.731072 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:42891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260929 20:20:29.732749 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:29.732171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:42891 (local address 127.0.0.1:56744)
0929 20:20:29.732480 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:29.732588 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:42891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
I20260929 20:20:29.737198 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 12336
I20260929 20:20:29.753005 12479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:29.737980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45680 (local address 127.27.185.67:45957)
0929 20:20:29.738182 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:29.738188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:29.738263 (+ 75us) server_negotiation.cc:407] Connection header received
0929 20:20:29.738486 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:29.738492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:29.738659 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:29.738849 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:20:29.740437 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.741327 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.742768 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:29.743245 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:29.743774 (+ 529us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:29.746672 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:20:29.746685 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:20:29.746701 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:20:29.746747 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:20:29.750011 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:29.750798 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:29.750805 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:29.750808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:29.750957 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:29.751347 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:29.751357 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:29.751360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:29.751761 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:20:29.752113 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:29.752415 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 20:20:29.752656 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 20:20:29.753759 12459 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:20:29.813568 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:29.813050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:42891 (local address 127.0.0.1:56752)
0929 20:20:29.813292 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:29.813385 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:42891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260929 20:20:29.814651 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:29.814264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:42891 (local address 127.0.0.1:56756)
0929 20:20:29.814466 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:29.814540 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:42891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 20:20:29.877172 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 12336
I20260929 20:20:29.898813 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 11983
I20260929 20:20:29.899983 12046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:20:30.014642 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:30.013992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:42891 (local address 127.0.0.1:56762)
0929 20:20:30.014241 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:30.014377 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:42891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 20:20:30.014758 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 11983
I20260929 20:20:50.037930 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:41609
--webserver_interface=127.27.185.126
--webserver_port=43017
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:43943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:41609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:20:50.377260 12511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:20:50.377584 12511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:20:50.377750 12511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:20:50.385095 12511 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 20:20:50.385206 12511 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 20:20:50.385288 12511 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 20:20:50.385380 12511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:20:50.385521 12511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:20:50.385629 12511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:20:50.385716 12511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:20:50.385800 12511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:20:50.395941 12511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:43943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:41609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:41609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=43017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:20:50.397416 12511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:20:50.398816 12511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:20:50.411284 12516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:20:50.411239 12519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:20:50.411486 12517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:20:50.412187 12511 server_base.cc:1034] running on GCE node
I20260929 20:20:50.413079 12511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:20:50.414847 12511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:20:50.416074 12511 hybrid_clock.cc:648] HybridClock initialized: now 1790713250416015 us; error 30 us; skew 500 ppm
I20260929 20:20:50.416404 12511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:20:50 dist-test-slave-4cx3 krb5kdc[11430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713250, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:20:50.424968 12511 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:20:50.426578 12511 webserver.cc:460] Webserver started at http://127.27.185.126:43017/ using document root <none> and password file <none>
I20260929 20:20:50.427311 12511 fs_manager.cc:359] Metadata directory not provided
I20260929 20:20:50.427466 12511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:20:50.433060 12511 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:20:50.435999 12526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:20:50.437583 12511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:20:50.437747 12511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "5562e2eff8df42f69e44202013a0c134"
format_stamp: "Formatted at 2026-09-29 20:20:04 on dist-test-slave-4cx3"
server_key: "581fafcbb6b9fb55652cf1fa48dbd301"
server_key_iv: "ea41d01177b62999a128ee64f88d6725"
server_key_version: "encryptionkey@0"
I20260929 20:20:50.438728 12511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:20:50.462383 12511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:20:50.466668 12511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:20:50.466907 12511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:20:50.486111 12511 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:41609
I20260929 20:20:50.486214 12577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:41609 every 8 connection(s)
I20260929 20:20:50.487711 12511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:20:50.493237 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 12511
I20260929 20:20:50.494555 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:42891
--local_ip_for_outbound_sockets=127.27.185.65
--tserver_master_addrs=127.27.185.126:41609
--webserver_port=39177
--webserver_interface=127.27.185.65
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:43943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:20:50.495796 12578 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:20:50.501251 12578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134: Bootstrap starting.
I20260929 20:20:50.526302 12578 log.cc:824] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134: Log is configured to *not* fsync() on all Append() calls
I20260929 20:20:50.542838 12578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:20:50.543537 12578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134: Bootstrap complete.
I20260929 20:20:50.554443 12578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5562e2eff8df42f69e44202013a0c134" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41609 } }
I20260929 20:20:50.555423 12578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5562e2eff8df42f69e44202013a0c134, State: Initialized, Role: FOLLOWER
I20260929 20:20:50.556177 12578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5562e2eff8df42f69e44202013a0c134" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41609 } }
I20260929 20:20:50.556428 12578 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:20:50.556547 12578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:20:50.556744 12578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [term 2 FOLLOWER]: Advancing to term 3
I20260929 20:20:50.560683 12578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5562e2eff8df42f69e44202013a0c134" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41609 } }
I20260929 20:20:50.561185 12578 leader_election.cc:302] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5562e2eff8df42f69e44202013a0c134; no voters:
I20260929 20:20:50.562073 12578 leader_election.cc:288] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 20:20:50.562213 12582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 20:20:50.563409 12582 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [term 3 LEADER]: Becoming Leader. State: Replica: 5562e2eff8df42f69e44202013a0c134, State: Running, Role: LEADER
I20260929 20:20:50.563978 12582 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5562e2eff8df42f69e44202013a0c134" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41609 } }
I20260929 20:20:50.564407 12578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:20:50.566550 12583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5562e2eff8df42f69e44202013a0c134" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5562e2eff8df42f69e44202013a0c134" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41609 } } }
I20260929 20:20:50.566972 12583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [sys.catalog]: This master's current role is: LEADER
I20260929 20:20:50.568847 12587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:20:50.568696 12584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5562e2eff8df42f69e44202013a0c134. Latest consensus state: current_term: 3 leader_uuid: "5562e2eff8df42f69e44202013a0c134" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5562e2eff8df42f69e44202013a0c134" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41609 } } }
I20260929 20:20:50.569103 12584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134 [sys.catalog]: This master's current role is: LEADER
I20260929 20:20:50.578758 12587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:20:50.580540 12587 catalog_manager.cc:1259] Loaded cluster ID: 890e623071744b2ea5e2b009ddb3566f
I20260929 20:20:50.580758 12587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:20:50.585110 12587 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:20:50.587685 12587 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134: Loaded TSK: 0
I20260929 20:20:50.588016 12587 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5562e2eff8df42f69e44202013a0c134: Loaded TSK: 1
F20260929 20:20:50.589352 12587 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f67419ae9ad google::LogMessage::Fail() at ??:0
@ 0x7f67419b188a google::LogMessage::SendToLog() at ??:0
@ 0x7f67419ae4de google::LogMessage::Flush() at ??:0
@ 0x7f67419afa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f67457127d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f674570173e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f67573e11c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f67573e0783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f675746c2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f675746c0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f67574b65e9 std::function<>::operator()() at ??:0
@ 0x7f67573e4cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f67573e2c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f675746a610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f675746a439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f67445fa345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f67445fd91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f67445fd799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f67445d8eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f67478b26db start_thread at ??:0
@ 0x7f674084861f clone at ??:0
W20260929 20:20:50.903097 12580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:20:50.903453 12580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:20:50.903597 12580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:20:50.911020 12580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:20:50.911187 12580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:20:50.911366 12580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:20:50.921465 12580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:43943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:42891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=39177
--tserver_master_addrs=127.27.185.126:41609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:20:50.922849 12580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:20:50.924366 12580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:20:50.938268 12625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:20:50.938215 12628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:20:50.938305 12626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:20:50.939054 12580 server_base.cc:1034] running on GCE node
I20260929 20:20:50.939730 12580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:20:50.940637 12580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:20:50.941870 12580 hybrid_clock.cc:648] HybridClock initialized: now 1790713250941831 us; error 52 us; skew 500 ppm
I20260929 20:20:50.942199 12580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:20:50 dist-test-slave-4cx3 krb5kdc[11430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713250, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:20:50.945446 12580 init.cc:377] Logged in from keytab as kudu/127.27.185.65@KRBTEST.COM (short username kudu)
I20260929 20:20:50.946961 12580 webserver.cc:460] Webserver started at http://127.27.185.65:39177/ using document root <none> and password file <none>
I20260929 20:20:50.947748 12580 fs_manager.cc:359] Metadata directory not provided
I20260929 20:20:50.947901 12580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:20:50.953359 12580 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 20:20:50.956498 12635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:20:50.957881 12580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:20:50.958070 12580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "1eaa1e9d171d400e913daf376fa7943f"
format_stamp: "Formatted at 2026-09-29 20:20:05 on dist-test-slave-4cx3"
server_key: "3f1f32d5303ba34e16b76822fd6592af"
server_key_iv: "e2f9e954fdfc11fd81a9650954cbc072"
server_key_version: "encryptionkey@0"
I20260929 20:20:50.959052 12580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:20:50.981164 12580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:20:50.985857 12580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:20:50.986153 12580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:20:50.987319 12580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:20:50.989334 12642 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260929 20:20:50.995021 12580 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 20:20:50.995159 12580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260929 20:20:50.995309 12580 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 20:20:50.998447 12580 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 20:20:50.998571 12580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260929 20:20:50.999080 12642 tablet_bootstrap.cc:492] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f: Bootstrap starting.
I20260929 20:20:51.008000 12642 log.cc:824] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:20:51.024032 12642 tablet_bootstrap.cc:492] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:20:51.025009 12642 tablet_bootstrap.cc:492] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f: Bootstrap complete.
I20260929 20:20:51.025635 12642 ts_tablet_manager.cc:1397] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f: Time spent bootstrapping tablet: real 0.027s user 0.023s sys 0.002s
I20260929 20:20:51.035599 12642 raft_consensus.cc:348] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38641 } } peers { permanent_uuid: "1eaa1e9d171d400e913daf376fa7943f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42891 } } peers { permanent_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45957 } }
I20260929 20:20:51.037317 12642 raft_consensus.cc:729] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eaa1e9d171d400e913daf376fa7943f, State: Initialized, Role: FOLLOWER
I20260929 20:20:51.038471 12642 consensus_queue.cc:260] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38641 } } peers { permanent_uuid: "1eaa1e9d171d400e913daf376fa7943f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42891 } } peers { permanent_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45957 } }
I20260929 20:20:51.040618 12642 ts_tablet_manager.cc:1428] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260929 20:20:51.040921 12580 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:42891
I20260929 20:20:51.040961 12750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:42891 every 8 connection(s)
I20260929 20:20:51.042161 12580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:20:51.046118 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 12580
I20260929 20:20:51.046996 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:38641
--local_ip_for_outbound_sockets=127.27.185.66
--tserver_master_addrs=127.27.185.126:41609
--webserver_port=45655
--webserver_interface=127.27.185.66
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:43943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:20:51.047721 12753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:51.045134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.65:33517)
0929 20:20:51.046124 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:51.046459 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":379,"threads_started":1}
W20260929 20:20:51.048355 12751 heartbeater.cc:644] Failed to heartbeat to 127.27.185.126:41609 (0 consecutive failures): Network error: Failed to ping master at 127.27.185.126:41609: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
I20260929 20:20:51.049335 12753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:51.048878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.65:48071)
0929 20:20:51.049068 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:51.049167 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 20:20:51.050313 12753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:51.049885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.65:52415)
0929 20:20:51.050058 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:51.050156 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 20:20:51.050582 12751 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 20:20:51.380734 12754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:20:51.381031 12754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:20:51.381196 12754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:20:51.388610 12754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:20:51.388723 12754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:20:51.388898 12754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:20:51.398773 12754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:43943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:38641
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=45655
--tserver_master_addrs=127.27.185.126:41609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:20:51.400219 12754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:20:51.401626 12754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:20:51.414853 12759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:20:51.414887 12762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:20:51.414966 12760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:20:51.416183 12754 server_base.cc:1034] running on GCE node
I20260929 20:20:51.416793 12754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:20:51.417747 12754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:20:51.418977 12754 hybrid_clock.cc:648] HybridClock initialized: now 1790713251418931 us; error 38 us; skew 500 ppm
I20260929 20:20:51.419298 12754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:20:51 dist-test-slave-4cx3 krb5kdc[11430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713251, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:20:51.422533 12754 init.cc:377] Logged in from keytab as kudu/127.27.185.66@KRBTEST.COM (short username kudu)
I20260929 20:20:51.424139 12754 webserver.cc:460] Webserver started at http://127.27.185.66:45655/ using document root <none> and password file <none>
I20260929 20:20:51.424867 12754 fs_manager.cc:359] Metadata directory not provided
I20260929 20:20:51.425019 12754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:20:51.430433 12754 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 20:20:51.433252 12769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:20:51.434569 12754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:20:51.434748 12754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a"
format_stamp: "Formatted at 2026-09-29 20:20:05 on dist-test-slave-4cx3"
server_key: "67af568a608460a635d7b4363445ee6a"
server_key_iv: "d3666038e879091cc8912c746c0d68c2"
server_key_version: "encryptionkey@0"
I20260929 20:20:51.435840 12754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:20:51.460855 12754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:20:51.465515 12754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:20:51.465809 12754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:20:51.466863 12754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:20:51.468319 12754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:20:51.468447 12754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:20:51.468588 12754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:20:51.468677 12754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:20:51.502585 12754 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:38641
I20260929 20:20:51.502643 12881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:38641 every 8 connection(s)
I20260929 20:20:51.504113 12754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:20:51.508966 12884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:51.507024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.66:43515)
0929 20:20:51.507784 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:51.508039 (+ 255us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":324,"threads_started":1}
W20260929 20:20:51.509616 12882 heartbeater.cc:644] Failed to heartbeat to 127.27.185.126:41609 (0 consecutive failures): Network error: Failed to ping master at 127.27.185.126:41609: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
I20260929 20:20:51.510397 12884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:51.510062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.66:33647)
0929 20:20:51.510216 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:51.510292 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 20:20:51.511195 12884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:51.510872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.66:39405)
0929 20:20:51.511006 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:51.511092 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 20:20:51.511473 12882 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 20:20:51.512624 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 12754
I20260929 20:20:51.513221 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:45957
--local_ip_for_outbound_sockets=127.27.185.67
--tserver_master_addrs=127.27.185.126:41609
--webserver_port=35807
--webserver_interface=127.27.185.67
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:43943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:20:51.844604 12885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:20:51.844875 12885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:20:51.844973 12885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:20:51.852186 12885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:20:51.852309 12885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:20:51.852427 12885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:20:51.862353 12885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:43943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:45957
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=35807
--tserver_master_addrs=127.27.185.126:41609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:20:51.863816 12885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:20:51.865052 12885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:20:51.878091 12890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:20:51.878187 12891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:20:51.878525 12893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:20:51.879033 12885 server_base.cc:1034] running on GCE node
I20260929 20:20:51.879709 12885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:20:51.880652 12885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:20:51.881856 12885 hybrid_clock.cc:648] HybridClock initialized: now 1790713251881804 us; error 46 us; skew 500 ppm
I20260929 20:20:51.882176 12885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:20:51 dist-test-slave-4cx3 krb5kdc[11430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713251, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:20:51.885367 12885 init.cc:377] Logged in from keytab as kudu/127.27.185.67@KRBTEST.COM (short username kudu)
I20260929 20:20:51.886749 12885 webserver.cc:460] Webserver started at http://127.27.185.67:35807/ using document root <none> and password file <none>
I20260929 20:20:51.887492 12885 fs_manager.cc:359] Metadata directory not provided
I20260929 20:20:51.887634 12885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:20:51.893141 12885 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 20:20:51.896135 12900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:20:51.897675 12885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:20:51.897862 12885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2"
format_stamp: "Formatted at 2026-09-29 20:20:06 on dist-test-slave-4cx3"
server_key: "5aebf81f95b9b81b6ed0a11724b8569a"
server_key_iv: "fcb66ae6f8487cbc9808933eea0c13a0"
server_key_version: "encryptionkey@0"
I20260929 20:20:51.898957 12885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:20:51.918748 12885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:20:51.923635 12885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:20:51.923936 12885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:20:51.924963 12885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:20:51.926287 12885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:20:51.926424 12885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:20:51.926573 12885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:20:51.926659 12885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:20:51.961154 12885 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:45957
I20260929 20:20:51.961190 13012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:45957 every 8 connection(s)
I20260929 20:20:51.962752 12885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:20:51.967741 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 12885
I20260929 20:20:51.968554 13015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:51.966040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.67:57371)
0929 20:20:51.966821 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:51.967067 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":331,"threads_started":1}
W20260929 20:20:51.969439 13013 heartbeater.cc:644] Failed to heartbeat to 127.27.185.126:41609 (0 consecutive failures): Network error: Failed to ping master at 127.27.185.126:41609: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
I20260929 20:20:51.970287 13015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:51.969942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.67:55553)
0929 20:20:51.970099 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:51.970177 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 20:20:51.971153 13015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:51.970797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.67:41755)
0929 20:20:51.970976 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:51.971049 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 20:20:51.971431 13013 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 20:20:52.051870 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:52.051094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.65:60571)
0929 20:20:52.051580 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:52.051660 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":201,"threads_started":1}
I20260929 20:20:52.370303 13018 raft_consensus.cc:482] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:20:52.370600 13018 raft_consensus.cc:504] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38641 } } peers { permanent_uuid: "1eaa1e9d171d400e913daf376fa7943f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42891 } } peers { permanent_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45957 } }
I20260929 20:20:52.372460 13018 leader_election.cc:288] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0e4a8bb2f91540cd9e2c15bb3408f43a (127.27.185.66:38641), ccce56dac1a84ed391bdc11b7dd6a1d2 (127.27.185.67:45957)
Sep 29 20:20:52 dist-test-slave-4cx3 krb5kdc[11430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713250, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for kudu/127.27.185.67@KRBTEST.COM
Sep 29 20:20:52 dist-test-slave-4cx3 krb5kdc[11430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713250, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for kudu/127.27.185.66@KRBTEST.COM
I20260929 20:20:52.395078 13019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:52.372898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:45957 (local address 127.27.185.65:43381)
0929 20:20:52.376007 (+ 3109us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:52.376028 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:20:52.376155 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:20:52.378641 (+ 2486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:20:52.378645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:20:52.379141 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:20:52.379608 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:52.379629 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:52.381234 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:52.381238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:52.382756 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:52.382769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:52.383888 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:52.383895 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:52.384192 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:52.384966 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:20:52.384989 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:20:52.387708 (+ 2719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:20:52.391473 (+ 3765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:20:52.391482 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:20:52.391499 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:20:52.392041 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:20:52.392551 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:20:52.392562 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:20:52.392565 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:20:52.392744 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:20:52.393672 (+ 928us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:20:52.393681 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:20:52.394011 (+ 330us) client_negotiation.cc:769] Sending connection context
0929 20:20:52.394380 (+ 369us) client_negotiation.cc:240] Negotiation successful
0929 20:20:52.394764 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2972,"mutex_wait_us":201,"spinlock_wait_cycles":12032,"thread_start_us":2795,"threads_started":1}
I20260929 20:20:52.395602 13021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:52.373521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:43381 (local address 127.27.185.67:45957)
0929 20:20:52.375806 (+ 2285us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:52.375822 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:52.376128 (+ 306us) server_negotiation.cc:407] Connection header received
0929 20:20:52.377860 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:52.377870 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:52.378328 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:52.378672 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:20:52.379818 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:52.381024 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:52.382911 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:52.383658 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:52.383861 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:52.388040 (+ 4179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:20:52.388053 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:20:52.388074 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:20:52.388128 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:20:52.391281 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:52.392202 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:52.392211 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:52.392225 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:52.392436 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:52.392868 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:52.392874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:52.392878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:52.393542 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:20:52.393686 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:52.394710 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0929 20:20:52.395140 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2069,"thread_start_us":1855,"threads_started":1}
I20260929 20:20:52.396353 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:52.372812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:38641 (local address 127.27.185.65:49739)
0929 20:20:52.373149 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:52.373189 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 20:20:52.373807 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:20:52.377811 (+ 4004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:20:52.377824 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:20:52.378714 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:20:52.379608 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:52.379630 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:52.381237 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:52.381241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:52.382753 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:20:52.382767 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:20:52.383874 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:20:52.383881 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:20:52.384189 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:52.385015 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:20:52.385031 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:20:52.388101 (+ 3070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:20:52.393283 (+ 5182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:20:52.393286 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:20:52.393289 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:20:52.393672 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:20:52.394134 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:20:52.394138 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:20:52.394141 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:20:52.394292 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:20:52.395614 (+ 1322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:20:52.395619 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:20:52.395778 (+ 159us) client_negotiation.cc:769] Sending connection context
0929 20:20:52.395929 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 20:20:52.396118 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":82,"spinlock_wait_cycles":14592}
I20260929 20:20:52.397226 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:52.373406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:49739 (local address 127.27.185.66:38641)
0929 20:20:52.374718 (+ 1312us) server_negotiation.cc:206] Beginning negotiation
0929 20:20:52.374740 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 20:20:52.374790 (+ 50us) server_negotiation.cc:407] Connection header received
0929 20:20:52.376881 (+ 2091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:20:52.376893 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:20:52.377299 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:20:52.377574 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:20:52.379820 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:52.381006 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:52.382919 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:20:52.383647 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:20:52.383851 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:20:52.388392 (+ 4541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:20:52.388402 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:20:52.388423 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:20:52.388466 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:20:52.393159 (+ 4693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:52.393811 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:52.393817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:52.393822 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:52.393987 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:20:52.394407 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:20:52.394412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:20:52.394415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:20:52.395060 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:20:52.395518 (+ 458us) server_negotiation.cc:1035] Waiting for connection context
0929 20:20:52.396171 (+ 653us) server_negotiation.cc:299] Negotiation successful
0929 20:20:52.396944 (+ 773us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1085,"thread_start_us":923,"threads_started":1}
W20260929 20:20:52.399123 12639 leader_election.cc:341] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ccce56dac1a84ed391bdc11b7dd6a1d2 (127.27.185.67:45957): Not found: Tablet not found: 4b818ba1c5b34941ae41c09fd91eae9a
W20260929 20:20:52.400182 12638 leader_election.cc:341] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0e4a8bb2f91540cd9e2c15bb3408f43a (127.27.185.66:38641): Not found: Tablet not found: 4b818ba1c5b34941ae41c09fd91eae9a
I20260929 20:20:52.400348 12638 leader_election.cc:302] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eaa1e9d171d400e913daf376fa7943f; no voters: 0e4a8bb2f91540cd9e2c15bb3408f43a, ccce56dac1a84ed391bdc11b7dd6a1d2
I20260929 20:20:52.400760 13018 raft_consensus.cc:2726] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 20:20:52.513299 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:52.512082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.66:33209)
0929 20:20:52.512704 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:52.512897 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":312,"threads_started":1}
I20260929 20:20:52.972714 13023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:52.971947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.67:47509)
0929 20:20:52.972394 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:52.972505 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":213,"threads_started":1}
I20260929 20:20:53.053607 13024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:53.052809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.65:50507)
0929 20:20:53.053272 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:53.053426 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":231,"threads_started":1}
I20260929 20:20:53.515045 13025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:53.514236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.66:45949)
0929 20:20:53.514797 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:53.514884 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":353,"threads_started":1}
I20260929 20:20:53.974435 13026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:53.973655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.67:44989)
0929 20:20:53.974132 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:53.974230 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":254,"threads_started":1}
I20260929 20:20:54.055312 13027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:54.054579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.65:48563)
0929 20:20:54.055047 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:54.055137 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":226,"threads_started":1}
I20260929 20:20:54.326411 13028 raft_consensus.cc:482] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:20:54.326653 13028 raft_consensus.cc:504] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38641 } } peers { permanent_uuid: "1eaa1e9d171d400e913daf376fa7943f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42891 } } peers { permanent_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45957 } }
I20260929 20:20:54.327325 13028 leader_election.cc:288] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0e4a8bb2f91540cd9e2c15bb3408f43a (127.27.185.66:38641), ccce56dac1a84ed391bdc11b7dd6a1d2 (127.27.185.67:45957)
W20260929 20:20:54.328439 12638 leader_election.cc:341] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0e4a8bb2f91540cd9e2c15bb3408f43a (127.27.185.66:38641): Not found: Tablet not found: 4b818ba1c5b34941ae41c09fd91eae9a
W20260929 20:20:54.328621 12639 leader_election.cc:341] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ccce56dac1a84ed391bdc11b7dd6a1d2 (127.27.185.67:45957): Not found: Tablet not found: 4b818ba1c5b34941ae41c09fd91eae9a
I20260929 20:20:54.328795 12639 leader_election.cc:302] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eaa1e9d171d400e913daf376fa7943f; no voters: 0e4a8bb2f91540cd9e2c15bb3408f43a, ccce56dac1a84ed391bdc11b7dd6a1d2
I20260929 20:20:54.329116 13028 raft_consensus.cc:2726] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 20:20:54.517020 13029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:54.516063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.66:46759)
0929 20:20:54.516657 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:54.516773 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":273,"threads_started":1}
I20260929 20:20:54.976327 13030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:54.975571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.67:54515)
0929 20:20:54.976070 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:54.976168 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":263,"threads_started":1}
I20260929 20:20:55.057183 13031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:55.056482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.65:57803)
0929 20:20:55.056929 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:55.057019 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":233,"threads_started":1}
I20260929 20:20:55.518643 13032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:55.517958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.66:50073)
0929 20:20:55.518377 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:55.518454 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":215,"threads_started":1}
I20260929 20:20:55.978161 13033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:55.977352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.67:41335)
0929 20:20:55.977853 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:55.977950 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":240,"threads_started":1}
I20260929 20:20:56.059231 13034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:56.058388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.65:35721)
0929 20:20:56.058948 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:56.059046 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":247,"threads_started":1}
I20260929 20:20:56.520428 13035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:56.519673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.66:35939)
0929 20:20:56.520124 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:56.520254 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":248,"threads_started":1}
I20260929 20:20:56.980110 13036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:56.979312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.67:57573)
0929 20:20:56.979834 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:56.979952 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":325,"threads_started":1}
I20260929 20:20:57.061118 13037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:57.060403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.65:60841)
0929 20:20:57.060844 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:57.060943 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":225,"threads_started":1}
I20260929 20:20:57.118584 13038 raft_consensus.cc:482] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:20:57.118903 13038 raft_consensus.cc:504] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38641 } } peers { permanent_uuid: "1eaa1e9d171d400e913daf376fa7943f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42891 } } peers { permanent_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45957 } }
I20260929 20:20:57.119696 13038 leader_election.cc:288] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0e4a8bb2f91540cd9e2c15bb3408f43a (127.27.185.66:38641), ccce56dac1a84ed391bdc11b7dd6a1d2 (127.27.185.67:45957)
W20260929 20:20:57.120683 12638 leader_election.cc:341] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0e4a8bb2f91540cd9e2c15bb3408f43a (127.27.185.66:38641): Not found: Tablet not found: 4b818ba1c5b34941ae41c09fd91eae9a
W20260929 20:20:57.120909 12639 leader_election.cc:341] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ccce56dac1a84ed391bdc11b7dd6a1d2 (127.27.185.67:45957): Not found: Tablet not found: 4b818ba1c5b34941ae41c09fd91eae9a
I20260929 20:20:57.121073 12639 leader_election.cc:302] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eaa1e9d171d400e913daf376fa7943f; no voters: 0e4a8bb2f91540cd9e2c15bb3408f43a, ccce56dac1a84ed391bdc11b7dd6a1d2
I20260929 20:20:57.121534 13038 raft_consensus.cc:2726] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 20:20:57.522137 13039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:57.521309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.66:60573)
0929 20:20:57.521774 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:57.521886 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":237,"threads_started":1}
I20260929 20:20:57.981927 13040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:57.981159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.67:38113)
0929 20:20:57.981657 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:57.981749 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":246,"threads_started":1}
I20260929 20:20:58.063050 13041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:58.062116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.65:55285)
0929 20:20:58.062756 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:58.062854 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":436,"threads_started":1}
I20260929 20:20:58.524133 13042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:58.523199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.66:42341)
0929 20:20:58.523838 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:58.523929 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":313,"threads_started":1}
I20260929 20:20:58.983814 13043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:58.982949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.67:60413)
0929 20:20:58.983511 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:58.983601 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":229,"threads_started":1}
I20260929 20:20:59.064750 13044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:59.064032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.65:45873)
0929 20:20:59.064502 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:59.064592 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":238,"threads_started":1}
I20260929 20:20:59.525799 13045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:59.525116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.66:43465)
0929 20:20:59.525517 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:59.525610 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":207,"threads_started":1}
I20260929 20:20:59.985611 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:20:59.984856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.67:38355)
0929 20:20:59.985337 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 20:20:59.985431 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":276,"threads_started":1}
I20260929 20:21:00.066581 13047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:00.065667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.65:41119)
0929 20:21:00.066180 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:00.066308 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":286,"threads_started":1}
I20260929 20:21:00.527648 13048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:00.526847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.66:50883)
0929 20:21:00.527333 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:00.527485 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":236,"threads_started":1}
I20260929 20:21:00.987336 13049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:00.986613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.67:35831)
0929 20:21:00.987091 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:00.987175 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":231,"threads_started":1}
I20260929 20:21:01.068439 13050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:01.067726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.65:34825)
0929 20:21:01.068196 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:01.068282 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":248,"threads_started":1}
I20260929 20:21:01.529357 13051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:01.528607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.66:42239)
0929 20:21:01.529050 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:01.529142 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":246,"threads_started":1}
I20260929 20:21:01.545310 13052 raft_consensus.cc:482] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:21:01.545514 13052 raft_consensus.cc:504] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38641 } } peers { permanent_uuid: "1eaa1e9d171d400e913daf376fa7943f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42891 } } peers { permanent_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45957 } }
I20260929 20:21:01.546136 13052 leader_election.cc:288] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0e4a8bb2f91540cd9e2c15bb3408f43a (127.27.185.66:38641), ccce56dac1a84ed391bdc11b7dd6a1d2 (127.27.185.67:45957)
W20260929 20:21:01.547079 12638 leader_election.cc:341] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0e4a8bb2f91540cd9e2c15bb3408f43a (127.27.185.66:38641): Not found: Tablet not found: 4b818ba1c5b34941ae41c09fd91eae9a
W20260929 20:21:01.547289 12639 leader_election.cc:341] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ccce56dac1a84ed391bdc11b7dd6a1d2 (127.27.185.67:45957): Not found: Tablet not found: 4b818ba1c5b34941ae41c09fd91eae9a
I20260929 20:21:01.547478 12639 leader_election.cc:302] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eaa1e9d171d400e913daf376fa7943f; no voters: 0e4a8bb2f91540cd9e2c15bb3408f43a, ccce56dac1a84ed391bdc11b7dd6a1d2
I20260929 20:21:01.547776 13052 raft_consensus.cc:2726] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 20:21:01.989400 13053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:01.988499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.67:37803)
0929 20:21:01.989047 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:01.989164 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":311,"threads_started":1}
I20260929 20:21:02.070204 13054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:02.069507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.65:51133)
0929 20:21:02.069929 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:02.070018 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":225,"threads_started":1}
I20260929 20:21:02.531095 13055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:02.530348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.66:50863)
0929 20:21:02.530820 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:02.530911 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":232,"threads_started":1}
I20260929 20:21:02.991091 13056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:02.990295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.67:59401)
0929 20:21:02.990809 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:02.990909 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":267,"threads_started":1}
I20260929 20:21:03.071980 13057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:03.071205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.65:58331)
0929 20:21:03.071719 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:03.071818 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":308,"threads_started":1}
I20260929 20:21:03.533206 13058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:03.532265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.66:38147)
0929 20:21:03.532839 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:03.532948 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":296,"threads_started":1}
I20260929 20:21:03.993157 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:03.992189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.67:37759)
0929 20:21:03.992787 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:03.992907 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":289,"threads_started":1}
I20260929 20:21:04.073601 13060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:04.072875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.65:33395)
0929 20:21:04.073336 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:04.073421 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":254,"threads_started":1}
I20260929 20:21:04.534950 13061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:04.534198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.66:44827)
0929 20:21:04.534685 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:04.534781 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":221,"threads_started":1}
I20260929 20:21:04.994897 13062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:04.994176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.67:37215)
0929 20:21:04.994640 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:04.994733 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":230,"threads_started":1}
I20260929 20:21:05.075590 13063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:05.074706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.65:40281)
0929 20:21:05.075212 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:05.075306 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":265,"threads_started":1}
I20260929 20:21:05.536720 13064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:05.535953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.66:49085)
0929 20:21:05.536406 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:05.536488 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":228,"threads_started":1}
I20260929 20:21:05.996605 13065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:05.995862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.67:50247)
0929 20:21:05.996308 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:05.996396 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":228,"threads_started":1}
I20260929 20:21:06.077696 13066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:06.076909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.65:45741)
0929 20:21:06.077411 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:06.077504 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":251,"threads_started":1}
I20260929 20:21:06.226984 13067 raft_consensus.cc:482] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:21:06.227233 13067 raft_consensus.cc:504] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4a8bb2f91540cd9e2c15bb3408f43a" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 38641 } } peers { permanent_uuid: "1eaa1e9d171d400e913daf376fa7943f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 42891 } } peers { permanent_uuid: "ccce56dac1a84ed391bdc11b7dd6a1d2" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45957 } }
I20260929 20:21:06.228142 13067 leader_election.cc:288] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0e4a8bb2f91540cd9e2c15bb3408f43a (127.27.185.66:38641), ccce56dac1a84ed391bdc11b7dd6a1d2 (127.27.185.67:45957)
W20260929 20:21:06.229156 12639 leader_election.cc:341] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ccce56dac1a84ed391bdc11b7dd6a1d2 (127.27.185.67:45957): Not found: Tablet not found: 4b818ba1c5b34941ae41c09fd91eae9a
W20260929 20:21:06.229378 12638 leader_election.cc:341] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0e4a8bb2f91540cd9e2c15bb3408f43a (127.27.185.66:38641): Not found: Tablet not found: 4b818ba1c5b34941ae41c09fd91eae9a
I20260929 20:21:06.229583 12638 leader_election.cc:302] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eaa1e9d171d400e913daf376fa7943f; no voters: 0e4a8bb2f91540cd9e2c15bb3408f43a, ccce56dac1a84ed391bdc11b7dd6a1d2
I20260929 20:21:06.230000 13067 raft_consensus.cc:2726] T 4b818ba1c5b34941ae41c09fd91eae9a P 1eaa1e9d171d400e913daf376fa7943f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 20:21:06.538614 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:06.537857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.66:33697)
0929 20:21:06.538324 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:06.538428 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":244,"threads_started":1}
I20260929 20:21:06.971113 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 12580
I20260929 20:21:06.986563 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:06.972526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57868 (local address 127.27.185.65:42891)
0929 20:21:06.973000 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:06.973008 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:06.973116 (+ 108us) server_negotiation.cc:407] Connection header received
0929 20:21:06.973271 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:06.973278 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:06.973516 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:06.973656 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:06.975068 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:06.976147 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:06.977417 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:06.977995 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:06.978116 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:06.980835 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:06.980858 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:06.980874 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:06.980915 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:06.983863 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:06.984569 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:06.984578 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:06.984580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:06.984720 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:06.985077 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:06.985080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:06.985082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:06.985458 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:06.985716 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:06.985953 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 20:21:06.986187 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":200,"threads_started":1}
I20260929 20:21:06.987977 12745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:06.998682 13071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:06.997826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.67:48845)
0929 20:21:06.998385 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:06.998519 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":300,"threads_started":1}
I20260929 20:21:07.088284 13072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:07.084264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41609 (local address 127.27.185.65:59993)
0929 20:21:07.087871 (+ 3607us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:07.087997 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.126:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3493,"thread_start_us":3348,"threads_started":1}
I20260929 20:21:07.088929 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 12580
I20260929 20:21:07.104702 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 12754
I20260929 20:21:07.118187 13073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:07.105487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52940 (local address 127.27.185.66:38641)
0929 20:21:07.106002 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:07.106006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:07.106023 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:21:07.106087 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:07.106090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:07.106153 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:07.106284 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:07.107712 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:07.108463 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:07.109680 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:07.110083 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:07.110178 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:07.112911 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:07.112917 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:07.112920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:07.112952 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:07.115870 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:07.116533 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:07.116548 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:07.116551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:07.116688 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:07.117001 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:07.117004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:07.117006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:07.117286 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:07.117498 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:07.117723 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 20:21:07.117898 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":285,"threads_started":1}
I20260929 20:21:07.118909 12877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:07.205096 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 12754
I20260929 20:21:07.223615 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 12885
I20260929 20:21:07.237326 13075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:07.224477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46608 (local address 127.27.185.67:45957)
0929 20:21:07.225081 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:07.225086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:07.225104 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:21:07.225164 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:07.225168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:07.225238 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:07.225333 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:07.226824 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:07.227807 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:07.229056 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:07.229483 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:07.229579 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:07.232439 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:07.232444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:07.232447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:07.232478 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:07.235011 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:07.235742 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:07.235746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:07.235748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:07.235879 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:07.236190 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:07.236193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:07.236195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:07.236444 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:07.236661 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:07.236877 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 20:21:07.237045 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":316,"threads_started":1}
I20260929 20:21:07.238078 13008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:07.324903 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 12885
2026-09-29T20:21:07Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64663 ms)
[----------] 30 tests from SecurityITest (300373 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:21:07 dist-test-slave-4cx3 krb5kdc[13079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:21:07 dist-test-slave-4cx3 krb5kdc[13079](info): set up 2 sockets
Sep 29 20:21:07 dist-test-slave-4cx3 krb5kdc[13079](info): commencing operation
krb5kdc: starting...
W20260929 20:21:08.446642 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:21:08 dist-test-slave-4cx3 krb5kdc[13079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:21:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:21:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
I20260929 20:21:08.715378 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:37963
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:36215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:37963
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:21:09.060024 13092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:09.060360 13092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:09.060528 13092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:09.068285 13092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:21:09.068454 13092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:09.068552 13092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:09.068646 13092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:21:09.068734 13092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:21:09.078939 13092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:36215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:37963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:37963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:09.080446 13092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:09.081841 13092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:09.094408 13097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:09.094463 13098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:09.094380 13100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:09.095224 13092 server_base.cc:1034] running on GCE node
I20260929 20:21:09.096079 13092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:09.097575 13092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:09.098804 13092 hybrid_clock.cc:648] HybridClock initialized: now 1790713269098745 us; error 50 us; skew 500 ppm
I20260929 20:21:09.099182 13092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:21:09 dist-test-slave-4cx3 krb5kdc[13079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713269, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:21:09.102468 13092 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:21:09.103987 13092 webserver.cc:460] Webserver started at http://127.27.185.126:38815/ using document root <none> and password file <none>
I20260929 20:21:09.104667 13092 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:09.104770 13092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:09.105022 13092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:09.108068 13092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "fc13acdf37b54e57b0b8970532a976bb"
format_stamp: "Formatted at 2026-09-29 20:21:09 on dist-test-slave-4cx3"
server_key: "f61309aa466ffe1fa3557ab4f639d817"
server_key_iv: "60e76fbdd18da6c749a5981059529d4b"
server_key_version: "encryptionkey@0"
I20260929 20:21:09.108764 13092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "fc13acdf37b54e57b0b8970532a976bb"
format_stamp: "Formatted at 2026-09-29 20:21:09 on dist-test-slave-4cx3"
server_key: "f61309aa466ffe1fa3557ab4f639d817"
server_key_iv: "60e76fbdd18da6c749a5981059529d4b"
server_key_version: "encryptionkey@0"
I20260929 20:21:09.114192 13092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:21:09.117889 13107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:09.119299 13092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 20:21:09.119583 13092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "fc13acdf37b54e57b0b8970532a976bb"
format_stamp: "Formatted at 2026-09-29 20:21:09 on dist-test-slave-4cx3"
server_key: "f61309aa466ffe1fa3557ab4f639d817"
server_key_iv: "60e76fbdd18da6c749a5981059529d4b"
server_key_version: "encryptionkey@0"
I20260929 20:21:09.119784 13092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:09.132848 13092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:09.136807 13092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:09.137048 13092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:09.155821 13092 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:37963
I20260929 20:21:09.155846 13158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:37963 every 8 connection(s)
I20260929 20:21:09.157267 13092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:21:09.160620 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 13092
I20260929 20:21:09.160832 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:21:09.161302 13159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:09.161550 28389 external_mini_cluster.cc:1428] Setting key dc3923806c45d435897f509edc13f23d
I20260929 20:21:09.176131 13159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc13acdf37b54e57b0b8970532a976bb: Bootstrap starting.
Sep 29 20:21:09 dist-test-slave-4cx3 krb5kdc[13079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:21:09.180027 13159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc13acdf37b54e57b0b8970532a976bb: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:09.181149 13159 log.cc:824] T 00000000000000000000000000000000 P fc13acdf37b54e57b0b8970532a976bb: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:09.184058 13159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc13acdf37b54e57b0b8970532a976bb: No bootstrap required, opened a new log
I20260929 20:21:09.185725 13162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:09.163610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36440 (local address 127.27.185.126:37963)
0929 20:21:09.164485 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:09.164505 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:09.164572 (+ 67us) server_negotiation.cc:407] Connection header received
0929 20:21:09.166519 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:09.166557 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:09.167085 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:09.167577 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:09.168874 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:09.170890 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:09.172344 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:09.173486 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:09.173690 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:09.178339 (+ 4649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:09.178348 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:09.178363 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:09.178405 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:09.181687 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:09.182425 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:09.182433 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:09.182439 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:09.182600 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:09.182983 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:09.182987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:09.182989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:09.183568 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:09.183881 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:09.184327 (+ 446us) server_negotiation.cc:299] Negotiation successful
0929 20:21:09.184772 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":342,"threads_started":1}
I20260929 20:21:09.192004 13159 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc13acdf37b54e57b0b8970532a976bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc13acdf37b54e57b0b8970532a976bb" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 37963 } }
I20260929 20:21:09.192493 13159 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc13acdf37b54e57b0b8970532a976bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:09.192626 13159 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc13acdf37b54e57b0b8970532a976bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc13acdf37b54e57b0b8970532a976bb, State: Initialized, Role: FOLLOWER
I20260929 20:21:09.193243 13159 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc13acdf37b54e57b0b8970532a976bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc13acdf37b54e57b0b8970532a976bb" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 37963 } }
I20260929 20:21:09.193451 13159 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc13acdf37b54e57b0b8970532a976bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:21:09.193594 13159 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc13acdf37b54e57b0b8970532a976bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:21:09.193768 13159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc13acdf37b54e57b0b8970532a976bb [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:09.196017 13159 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc13acdf37b54e57b0b8970532a976bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc13acdf37b54e57b0b8970532a976bb" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 37963 } }
I20260929 20:21:09.196527 13159 leader_election.cc:302] T 00000000000000000000000000000000 P fc13acdf37b54e57b0b8970532a976bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc13acdf37b54e57b0b8970532a976bb; no voters:
I20260929 20:21:09.197093 13159 leader_election.cc:288] T 00000000000000000000000000000000 P fc13acdf37b54e57b0b8970532a976bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:21:09.197220 13164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc13acdf37b54e57b0b8970532a976bb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:21:09.197928 13164 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc13acdf37b54e57b0b8970532a976bb [term 1 LEADER]: Becoming Leader. State: Replica: fc13acdf37b54e57b0b8970532a976bb, State: Running, Role: LEADER
I20260929 20:21:09.198504 13164 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc13acdf37b54e57b0b8970532a976bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc13acdf37b54e57b0b8970532a976bb" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 37963 } }
I20260929 20:21:09.199200 13159 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc13acdf37b54e57b0b8970532a976bb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:21:09.201653 13165 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc13acdf37b54e57b0b8970532a976bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc13acdf37b54e57b0b8970532a976bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc13acdf37b54e57b0b8970532a976bb" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 37963 } } }
I20260929 20:21:09.202126 13165 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc13acdf37b54e57b0b8970532a976bb [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:09.202677 13166 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc13acdf37b54e57b0b8970532a976bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc13acdf37b54e57b0b8970532a976bb. Latest consensus state: current_term: 1 leader_uuid: "fc13acdf37b54e57b0b8970532a976bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc13acdf37b54e57b0b8970532a976bb" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 37963 } } }
I20260929 20:21:09.203016 13166 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc13acdf37b54e57b0b8970532a976bb [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:09.203791 13171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:21:09.209477 13171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:21:09.219101 13171 catalog_manager.cc:1347] Generated new cluster ID: 25d7c506eb99468aa5e4c9abf97b7cdb
I20260929 20:21:09.219223 13171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:21:09.240947 13171 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:21:09.242344 13171 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:21:09.249985 13171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc13acdf37b54e57b0b8970532a976bb: Generated new TSK 0
I20260929 20:21:09.250838 13171 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
I20260929 20:21:09.318980 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:0
--local_ip_for_outbound_sockets=127.27.185.65
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:37963
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:36215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:21:09.668270 13185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:09.668565 13185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:09.668660 13185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:09.676162 13185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:09.676321 13185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:09.676546 13185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:21:09.687019 13185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:36215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:37963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:09.688378 13185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:09.689754 13185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:09.703574 13193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:09.703656 13190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:09.703631 13191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:09.704829 13185 server_base.cc:1034] running on GCE node
I20260929 20:21:09.705476 13185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:09.706456 13185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:09.707649 13185 hybrid_clock.cc:648] HybridClock initialized: now 1790713269707588 us; error 57 us; skew 500 ppm
I20260929 20:21:09.707964 13185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:21:09 dist-test-slave-4cx3 krb5kdc[13079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713269, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:21:09.711114 13185 init.cc:377] Logged in from keytab as kudu/127.27.185.65@KRBTEST.COM (short username kudu)
I20260929 20:21:09.712661 13185 webserver.cc:460] Webserver started at http://127.27.185.65:39991/ using document root <none> and password file <none>
I20260929 20:21:09.713361 13185 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:09.713505 13185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:09.713819 13185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:09.716786 13185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-0/data/instance:
uuid: "cab0258603694a59b6bb8466b626d11e"
format_stamp: "Formatted at 2026-09-29 20:21:09 on dist-test-slave-4cx3"
server_key: "779dfa2d270b31d8a6f1649d84cdd7cb"
server_key_iv: "f03cc897875428491e64a556577dc239"
server_key_version: "encryptionkey@0"
I20260929 20:21:09.717561 13185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance:
uuid: "cab0258603694a59b6bb8466b626d11e"
format_stamp: "Formatted at 2026-09-29 20:21:09 on dist-test-slave-4cx3"
server_key: "779dfa2d270b31d8a6f1649d84cdd7cb"
server_key_iv: "f03cc897875428491e64a556577dc239"
server_key_version: "encryptionkey@0"
I20260929 20:21:09.722831 13185 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 20:21:09.726509 13200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:09.728092 13185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:21:09.728228 13185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "cab0258603694a59b6bb8466b626d11e"
format_stamp: "Formatted at 2026-09-29 20:21:09 on dist-test-slave-4cx3"
server_key: "779dfa2d270b31d8a6f1649d84cdd7cb"
server_key_iv: "f03cc897875428491e64a556577dc239"
server_key_version: "encryptionkey@0"
I20260929 20:21:09.728411 13185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:09.752279 13185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:09.756940 13185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:09.757258 13185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:09.758275 13185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:21:09.759701 13185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:21:09.759778 13185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:09.759943 13185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:21:09.760021 13185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:09.794137 13185 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:44973
I20260929 20:21:09.794234 13312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:44973 every 8 connection(s)
I20260929 20:21:09.795532 13185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:21:09.804824 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 13185
I20260929 20:21:09.805039 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance
I20260929 20:21:09.805723 28389 external_mini_cluster.cc:1428] Setting key 5db7d0070d211bf28cdb4eb7aee7fde1
Sep 29 20:21:09 dist-test-slave-4cx3 krb5kdc[13079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713269, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:21:09.817296 13315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:09.798144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:51073 (local address 127.27.185.126:37963)
0929 20:21:09.798610 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:09.798616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:09.799740 (+ 1124us) server_negotiation.cc:407] Connection header received
0929 20:21:09.801646 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:09.801651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:09.801728 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:09.801825 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:09.803296 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:09.804096 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:09.805883 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:09.806459 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:09.806599 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:09.810570 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:09.810579 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:09.810586 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:09.810632 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:09.814175 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:09.814963 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:09.814971 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:09.814976 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:09.815124 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:09.815581 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:09.815589 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:09.815594 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:09.815977 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:09.816139 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:09.816833 (+ 694us) server_negotiation.cc:299] Negotiation successful
0929 20:21:09.817039 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":213,"threads_started":1}
I20260929 20:21:09.818785 13316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:09.798748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:37963 (local address 127.27.185.65:51073)
0929 20:21:09.799569 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:09.799629 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:09.801381 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:09.802024 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:09.802032 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:09.802574 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:21:09.803146 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:09.803163 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:09.804266 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:09.804270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:09.805720 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:09.805732 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:09.806700 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:09.806710 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:09.806956 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:09.807705 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:21:09.807734 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:21:09.810322 (+ 2588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:21:09.814390 (+ 4068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:21:09.814397 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:21:09.814409 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:21:09.814841 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:21:09.815232 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:21:09.815235 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:21:09.815237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:21:09.815477 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:21:09.816096 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:21:09.816102 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:21:09.816440 (+ 338us) client_negotiation.cc:769] Sending connection context
0929 20:21:09.816664 (+ 224us) client_negotiation.cc:240] Negotiation successful
0929 20:21:09.817259 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":323,"threads_started":1}
I20260929 20:21:09.820389 13313 heartbeater.cc:347] Connected to a master server at 127.27.185.126:37963
I20260929 20:21:09.820734 13313 heartbeater.cc:464] Registering TS with master...
I20260929 20:21:09.821436 13313 heartbeater.cc:511] Master 127.27.185.126:37963 requested a full tablet report, sending...
I20260929 20:21:09.823586 13124 ts_manager.cc:194] Registered new tserver with Master: cab0258603694a59b6bb8466b626d11e (127.27.185.65:44973)
I20260929 20:21:09.825171 13124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.65@KRBTEST.COM'} at 127.27.185.65:51073
WARNING: no policy specified for kudu/127.27.185.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
I20260929 20:21:09.863462 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:0
--local_ip_for_outbound_sockets=127.27.185.66
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:37963
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:36215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:21:10.203055 13319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:10.203361 13319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:10.203583 13319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:10.211177 13319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:10.211295 13319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:10.211475 13319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:21:10.221719 13319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:36215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:37963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:10.223198 13319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:10.224690 13319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:10.238466 13324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:10.238382 13327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:10.238430 13325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:10.239315 13319 server_base.cc:1034] running on GCE node
I20260929 20:21:10.240000 13319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:10.241012 13319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:10.242218 13319 hybrid_clock.cc:648] HybridClock initialized: now 1790713270242172 us; error 41 us; skew 500 ppm
I20260929 20:21:10.242549 13319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:21:10 dist-test-slave-4cx3 krb5kdc[13079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713270, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:21:10.245662 13319 init.cc:377] Logged in from keytab as kudu/127.27.185.66@KRBTEST.COM (short username kudu)
I20260929 20:21:10.247144 13319 webserver.cc:460] Webserver started at http://127.27.185.66:45885/ using document root <none> and password file <none>
I20260929 20:21:10.247905 13319 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:10.248059 13319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:10.248382 13319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:10.251478 13319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-1/data/instance:
uuid: "9ca6fddfdcb04539a7503b9b83d107e9"
format_stamp: "Formatted at 2026-09-29 20:21:10 on dist-test-slave-4cx3"
server_key: "f96bb74c7e81fd5ccf9b745a968b48b9"
server_key_iv: "70e9236beca192c82e765092b39e7644"
server_key_version: "encryptionkey@0"
I20260929 20:21:10.252301 13319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance:
uuid: "9ca6fddfdcb04539a7503b9b83d107e9"
format_stamp: "Formatted at 2026-09-29 20:21:10 on dist-test-slave-4cx3"
server_key: "f96bb74c7e81fd5ccf9b745a968b48b9"
server_key_iv: "70e9236beca192c82e765092b39e7644"
server_key_version: "encryptionkey@0"
I20260929 20:21:10.257891 13319 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:21:10.261612 13334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:10.263024 13319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:21:10.263211 13319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "9ca6fddfdcb04539a7503b9b83d107e9"
format_stamp: "Formatted at 2026-09-29 20:21:10 on dist-test-slave-4cx3"
server_key: "f96bb74c7e81fd5ccf9b745a968b48b9"
server_key_iv: "70e9236beca192c82e765092b39e7644"
server_key_version: "encryptionkey@0"
I20260929 20:21:10.263432 13319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:10.280675 13319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:10.284940 13319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:10.285260 13319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:10.286311 13319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:21:10.287881 13319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:21:10.287985 13319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:10.288141 13319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:21:10.288228 13319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:10.321920 13319 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:34183
I20260929 20:21:10.321969 13446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:34183 every 8 connection(s)
I20260929 20:21:10.323307 13319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:21:10.327967 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 13319
I20260929 20:21:10.328153 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance
I20260929 20:21:10.328599 28389 external_mini_cluster.cc:1428] Setting key d3419d6654abd776e5b15e70bca16293
Sep 29 20:21:10 dist-test-slave-4cx3 krb5kdc[13079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713270, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:21:10.344147 13449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:10.325876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:43395 (local address 127.27.185.126:37963)
0929 20:21:10.326307 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:10.326312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:10.327372 (+ 1060us) server_negotiation.cc:407] Connection header received
0929 20:21:10.329433 (+ 2061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:10.329438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:10.329515 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:10.329653 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:10.331303 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:10.332179 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:10.333816 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:10.334250 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:10.334376 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:10.338053 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:10.338058 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:10.338062 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:10.338094 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:10.340750 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:10.341471 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:10.341478 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:10.341481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:10.341634 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:10.342240 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:10.342246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:10.342249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:10.342564 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:10.342728 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:10.343501 (+ 773us) server_negotiation.cc:299] Negotiation successful
0929 20:21:10.343743 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":220,"threads_started":1}
I20260929 20:21:10.345263 13450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:10.326472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:37963 (local address 127.27.185.66:43395)
0929 20:21:10.327221 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:10.327262 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:10.329144 (+ 1882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:10.329893 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:10.329902 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:10.330572 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:21:10.331127 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:10.331145 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:10.332355 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:10.332359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:10.333668 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:10.333680 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:10.334477 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:10.334484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:10.334720 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:10.335455 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:21:10.335479 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:21:10.337813 (+ 2334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:21:10.340901 (+ 3088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:21:10.340911 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:21:10.340928 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:21:10.341352 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:21:10.341882 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:21:10.341887 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:21:10.341892 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:21:10.342115 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:21:10.342701 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:21:10.342710 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:21:10.343072 (+ 362us) client_negotiation.cc:769] Sending connection context
0929 20:21:10.343360 (+ 288us) client_negotiation.cc:240] Negotiation successful
0929 20:21:10.343970 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":263,"threads_started":1}
I20260929 20:21:10.346930 13447 heartbeater.cc:347] Connected to a master server at 127.27.185.126:37963
I20260929 20:21:10.347285 13447 heartbeater.cc:464] Registering TS with master...
I20260929 20:21:10.348277 13447 heartbeater.cc:511] Master 127.27.185.126:37963 requested a full tablet report, sending...
I20260929 20:21:10.350057 13124 ts_manager.cc:194] Registered new tserver with Master: 9ca6fddfdcb04539a7503b9b83d107e9 (127.27.185.66:34183)
I20260929 20:21:10.351331 13124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.66@KRBTEST.COM'} at 127.27.185.66:43395
WARNING: no policy specified for kudu/127.27.185.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
I20260929 20:21:10.386914 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:0
--local_ip_for_outbound_sockets=127.27.185.67
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:37963
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:36215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:21:10.728574 13453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:10.728953 13453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:10.729113 13453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:10.736546 13453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:10.736706 13453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:10.736886 13453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:21:10.746466 13453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:36215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:37963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:10.747822 13453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:10.749171 13453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:10.762678 13458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:10.762737 13459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:10.762871 13461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:10.763636 13453 server_base.cc:1034] running on GCE node
I20260929 20:21:10.764324 13453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:10.765291 13453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:10.766521 13453 hybrid_clock.cc:648] HybridClock initialized: now 1790713270766479 us; error 49 us; skew 500 ppm
I20260929 20:21:10.766908 13453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:21:10 dist-test-slave-4cx3 krb5kdc[13079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713270, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:21:10.770190 13453 init.cc:377] Logged in from keytab as kudu/127.27.185.67@KRBTEST.COM (short username kudu)
I20260929 20:21:10.771756 13453 webserver.cc:460] Webserver started at http://127.27.185.67:44487/ using document root <none> and password file <none>
I20260929 20:21:10.772466 13453 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:10.772621 13453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:10.772938 13453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:10.775925 13453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-2/data/instance:
uuid: "8a2eea50f0dc4e1885f6a634ffee5659"
format_stamp: "Formatted at 2026-09-29 20:21:10 on dist-test-slave-4cx3"
server_key: "e84df356fbefdce10a867b421d0e66f2"
server_key_iv: "227f44210497581aadf876e240256cbc"
server_key_version: "encryptionkey@0"
I20260929 20:21:10.776712 13453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance:
uuid: "8a2eea50f0dc4e1885f6a634ffee5659"
format_stamp: "Formatted at 2026-09-29 20:21:10 on dist-test-slave-4cx3"
server_key: "e84df356fbefdce10a867b421d0e66f2"
server_key_iv: "227f44210497581aadf876e240256cbc"
server_key_version: "encryptionkey@0"
I20260929 20:21:10.781973 13453 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:21:10.785552 13468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:10.786926 13453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:21:10.787106 13453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "8a2eea50f0dc4e1885f6a634ffee5659"
format_stamp: "Formatted at 2026-09-29 20:21:10 on dist-test-slave-4cx3"
server_key: "e84df356fbefdce10a867b421d0e66f2"
server_key_iv: "227f44210497581aadf876e240256cbc"
server_key_version: "encryptionkey@0"
I20260929 20:21:10.787313 13453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:10.809895 13453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:10.814455 13453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:10.814757 13453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:10.815766 13453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:21:10.817087 13453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:21:10.817195 13453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:10.817332 13453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:21:10.817400 13453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:10.828325 13313 heartbeater.cc:503] Master 127.27.185.126:37963 was elected leader, sending a full tablet report...
I20260929 20:21:10.851074 13453 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:45749
I20260929 20:21:10.851155 13580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:45749 every 8 connection(s)
I20260929 20:21:10.852442 13453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:21:10.861202 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 13453
I20260929 20:21:10.861395 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance
I20260929 20:21:10.861847 28389 external_mini_cluster.cc:1428] Setting key c267d97cd1c5f6cb20ac516837244cd8
Sep 29 20:21:10 dist-test-slave-4cx3 krb5kdc[13079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713270, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:21:10.872169 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:10.854828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:50145 (local address 127.27.185.126:37963)
0929 20:21:10.855276 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:10.855281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:10.856184 (+ 903us) server_negotiation.cc:407] Connection header received
0929 20:21:10.858059 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:10.858065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:10.858158 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:10.858276 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:10.859859 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:10.860638 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:10.862314 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:10.862888 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:10.863019 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:10.866661 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:10.866665 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:10.866669 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:10.866700 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:10.869285 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:10.869978 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:10.869982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:10.869985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:10.870107 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:10.870459 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:10.870463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:10.870465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:10.870722 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:10.870855 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:10.871711 (+ 856us) server_negotiation.cc:299] Negotiation successful
0929 20:21:10.871888 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":232,"threads_started":1}
I20260929 20:21:10.872922 13584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:10.855282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:37963 (local address 127.27.185.67:50145)
0929 20:21:10.856015 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:10.856066 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:10.857760 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:10.858521 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:10.858529 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:10.859064 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:21:10.859688 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:10.859710 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:10.860801 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:10.860805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:10.862173 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:10.862186 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:10.863131 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:10.863139 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:10.863368 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:10.864163 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:21:10.864187 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:21:10.866460 (+ 2273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:21:10.869422 (+ 2962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:21:10.869428 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:21:10.869441 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:21:10.869867 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:21:10.870194 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:21:10.870197 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:21:10.870200 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:21:10.870365 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:21:10.870827 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:21:10.870833 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:21:10.871139 (+ 306us) client_negotiation.cc:769] Sending connection context
0929 20:21:10.871355 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 20:21:10.871796 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":259,"threads_started":1}
I20260929 20:21:10.874454 13581 heartbeater.cc:347] Connected to a master server at 127.27.185.126:37963
I20260929 20:21:10.874759 13581 heartbeater.cc:464] Registering TS with master...
I20260929 20:21:10.875627 13581 heartbeater.cc:511] Master 127.27.185.126:37963 requested a full tablet report, sending...
I20260929 20:21:10.877243 13123 ts_manager.cc:194] Registered new tserver with Master: 8a2eea50f0dc4e1885f6a634ffee5659 (127.27.185.67:45749)
I20260929 20:21:10.878119 13123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.67@KRBTEST.COM'} at 127.27.185.67:50145
I20260929 20:21:10.889410 28389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:21:10.906823 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:10.894080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36452 (local address 127.27.185.126:37963)
0929 20:21:10.894269 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:10.894275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:10.894752 (+ 477us) server_negotiation.cc:407] Connection header received
0929 20:21:10.894884 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:10.894888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:10.894951 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:10.895031 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:10.896476 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:10.897189 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:10.898528 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:10.898931 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:10.899022 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:10.901622 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:10.901627 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:10.901630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:10.901663 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:10.904366 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:10.905050 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:10.905054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:10.905056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:10.905169 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:10.905484 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:10.905487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:10.905489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:10.905847 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:10.906032 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:10.906305 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 20:21:10.906474 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:21:10.910744 13123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:21:10.917698 13123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:21:10.954352 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:10.930506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58734 (local address 127.27.185.66:34183)
0929 20:21:10.942945 (+ 12439us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:10.942960 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:10.943986 (+ 1026us) server_negotiation.cc:407] Connection header received
0929 20:21:10.944409 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:10.944416 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:10.944735 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:10.944930 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:21:10.946140 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:10.948280 (+ 2140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:10.951563 (+ 3283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:10.952873 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:10.953019 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:10.953235 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:10.953858 (+ 623us) server_negotiation.cc:299] Negotiation successful
0929 20:21:10.954026 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12204,"thread_start_us":12462,"threads_started":1}
I20260929 20:21:10.954504 13595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:10.930080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:34183 (local address 127.0.0.1:58734)
0929 20:21:10.943819 (+ 13739us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:10.943858 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:10.944162 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:10.945266 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:10.945277 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:10.945319 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:21:10.945885 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:10.945902 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:10.948484 (+ 2582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:10.948492 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:10.951334 (+ 2842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:10.951346 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:10.953359 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:10.953365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:10.953580 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:10.953643 (+ 63us) client_negotiation.cc:769] Sending connection context
0929 20:21:10.953815 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 20:21:10.954022 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13466,"mutex_wait_us":2,"thread_start_us":13317,"threads_started":1}
I20260929 20:21:10.955375 13594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:10.930139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:44973 (local address 127.0.0.1:37600)
0929 20:21:10.944420 (+ 14281us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:10.944437 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:10.944548 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:10.945734 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:10.945742 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:10.945777 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:21:10.946354 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:10.946369 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:10.948606 (+ 2237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:10.948614 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:10.951632 (+ 3018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:10.951647 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:10.953757 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:10.953766 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:10.954032 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:10.954717 (+ 685us) client_negotiation.cc:769] Sending connection context
0929 20:21:10.954820 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 20:21:10.954970 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14150,"mutex_wait_us":783,"thread_start_us":13973,"threads_started":1}
I20260929 20:21:10.955417 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:10.930757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37600 (local address 127.27.185.65:44973)
0929 20:21:10.943676 (+ 12919us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:10.943688 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:10.944685 (+ 997us) server_negotiation.cc:407] Connection header received
0929 20:21:10.944819 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:10.944830 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:10.945212 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:10.945441 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:21:10.946599 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:10.948397 (+ 1798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:10.951861 (+ 3464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:10.953280 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:10.953409 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:10.953610 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:10.954889 (+ 1279us) server_negotiation.cc:299] Negotiation successful
0929 20:21:10.955070 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12642,"thread_start_us":12925,"threads_started":1}
I20260929 20:21:10.958205 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:10.944814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60516 (local address 127.27.185.67:45749)
0929 20:21:10.950935 (+ 6121us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:10.950944 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:10.950984 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:21:10.951077 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:10.951085 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:10.951296 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:10.951509 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:21:10.952152 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:10.953731 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:10.956199 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:10.957328 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:10.957430 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:10.957591 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:10.957850 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 20:21:10.957961 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5951,"thread_start_us":5816,"threads_started":1}
I20260929 20:21:10.958297 13598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:10.944535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:45749 (local address 127.0.0.1:60516)
0929 20:21:10.949749 (+ 5214us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:10.949763 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:10.949852 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:10.951497 (+ 1645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:10.951503 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:10.951538 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:21:10.951964 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:10.951976 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:10.953882 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:10.953889 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:10.956028 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:10.956046 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:10.957521 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:10.957527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:10.957686 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:10.957715 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 20:21:10.957811 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 20:21:10.957932 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5082,"mutex_wait_us":20,"thread_start_us":4943,"threads_started":1}
I20260929 20:21:10.958727 13382 tablet_service.cc:1467] Processing CreateTablet for tablet 583ba459d3c9497285b0b1cc0eb82595 (DEFAULT_TABLE table=test-table [id=5e39d755f93a4a56bd0c9465d6334101]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:21:10.960295 13382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 583ba459d3c9497285b0b1cc0eb82595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:10.962028 13516 tablet_service.cc:1467] Processing CreateTablet for tablet 583ba459d3c9497285b0b1cc0eb82595 (DEFAULT_TABLE table=test-table [id=5e39d755f93a4a56bd0c9465d6334101]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:21:10.962029 13248 tablet_service.cc:1467] Processing CreateTablet for tablet 583ba459d3c9497285b0b1cc0eb82595 (DEFAULT_TABLE table=test-table [id=5e39d755f93a4a56bd0c9465d6334101]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:21:10.963557 13516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 583ba459d3c9497285b0b1cc0eb82595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:10.963579 13248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 583ba459d3c9497285b0b1cc0eb82595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:10.971804 13600 tablet_bootstrap.cc:492] T 583ba459d3c9497285b0b1cc0eb82595 P 9ca6fddfdcb04539a7503b9b83d107e9: Bootstrap starting.
I20260929 20:21:10.974956 13600 tablet_bootstrap.cc:654] T 583ba459d3c9497285b0b1cc0eb82595 P 9ca6fddfdcb04539a7503b9b83d107e9: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:10.976094 13600 log.cc:824] T 583ba459d3c9497285b0b1cc0eb82595 P 9ca6fddfdcb04539a7503b9b83d107e9: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:10.978619 13602 tablet_bootstrap.cc:492] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e: Bootstrap starting.
I20260929 20:21:10.980245 13600 tablet_bootstrap.cc:492] T 583ba459d3c9497285b0b1cc0eb82595 P 9ca6fddfdcb04539a7503b9b83d107e9: No bootstrap required, opened a new log
I20260929 20:21:10.980742 13600 ts_tablet_manager.cc:1397] T 583ba459d3c9497285b0b1cc0eb82595 P 9ca6fddfdcb04539a7503b9b83d107e9: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 20:21:10.982391 13602 tablet_bootstrap.cc:654] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:10.983705 13602 log.cc:824] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:10.984181 13603 tablet_bootstrap.cc:492] T 583ba459d3c9497285b0b1cc0eb82595 P 8a2eea50f0dc4e1885f6a634ffee5659: Bootstrap starting.
I20260929 20:21:10.987205 13602 tablet_bootstrap.cc:492] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e: No bootstrap required, opened a new log
I20260929 20:21:10.987507 13603 tablet_bootstrap.cc:654] T 583ba459d3c9497285b0b1cc0eb82595 P 8a2eea50f0dc4e1885f6a634ffee5659: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:10.987551 13602 ts_tablet_manager.cc:1397] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 20:21:10.988468 13603 log.cc:824] T 583ba459d3c9497285b0b1cc0eb82595 P 8a2eea50f0dc4e1885f6a634ffee5659: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:10.991165 13603 tablet_bootstrap.cc:492] T 583ba459d3c9497285b0b1cc0eb82595 P 8a2eea50f0dc4e1885f6a634ffee5659: No bootstrap required, opened a new log
I20260929 20:21:10.991508 13603 ts_tablet_manager.cc:1397] T 583ba459d3c9497285b0b1cc0eb82595 P 8a2eea50f0dc4e1885f6a634ffee5659: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260929 20:21:10.991317 13600 raft_consensus.cc:348] T 583ba459d3c9497285b0b1cc0eb82595 P 9ca6fddfdcb04539a7503b9b83d107e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca6fddfdcb04539a7503b9b83d107e9" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 34183 } } peers { permanent_uuid: "cab0258603694a59b6bb8466b626d11e" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 44973 } } peers { permanent_uuid: "8a2eea50f0dc4e1885f6a634ffee5659" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45749 } }
I20260929 20:21:10.991798 13600 raft_consensus.cc:374] T 583ba459d3c9497285b0b1cc0eb82595 P 9ca6fddfdcb04539a7503b9b83d107e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:10.991935 13600 raft_consensus.cc:729] T 583ba459d3c9497285b0b1cc0eb82595 P 9ca6fddfdcb04539a7503b9b83d107e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca6fddfdcb04539a7503b9b83d107e9, State: Initialized, Role: FOLLOWER
I20260929 20:21:10.992921 13600 consensus_queue.cc:260] T 583ba459d3c9497285b0b1cc0eb82595 P 9ca6fddfdcb04539a7503b9b83d107e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca6fddfdcb04539a7503b9b83d107e9" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 34183 } } peers { permanent_uuid: "cab0258603694a59b6bb8466b626d11e" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 44973 } } peers { permanent_uuid: "8a2eea50f0dc4e1885f6a634ffee5659" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45749 } }
I20260929 20:21:10.994670 13447 heartbeater.cc:503] Master 127.27.185.126:37963 was elected leader, sending a full tablet report...
I20260929 20:21:10.995311 13600 ts_tablet_manager.cc:1428] T 583ba459d3c9497285b0b1cc0eb82595 P 9ca6fddfdcb04539a7503b9b83d107e9: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 20:21:10.998080 13602 raft_consensus.cc:348] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca6fddfdcb04539a7503b9b83d107e9" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 34183 } } peers { permanent_uuid: "cab0258603694a59b6bb8466b626d11e" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 44973 } } peers { permanent_uuid: "8a2eea50f0dc4e1885f6a634ffee5659" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45749 } }
I20260929 20:21:10.998440 13602 raft_consensus.cc:374] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:10.998555 13602 raft_consensus.cc:729] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab0258603694a59b6bb8466b626d11e, State: Initialized, Role: FOLLOWER
I20260929 20:21:10.999179 13602 consensus_queue.cc:260] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca6fddfdcb04539a7503b9b83d107e9" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 34183 } } peers { permanent_uuid: "cab0258603694a59b6bb8466b626d11e" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 44973 } } peers { permanent_uuid: "8a2eea50f0dc4e1885f6a634ffee5659" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45749 } }
I20260929 20:21:11.000988 13602 ts_tablet_manager.cc:1428] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260929 20:21:11.000970 13603 raft_consensus.cc:348] T 583ba459d3c9497285b0b1cc0eb82595 P 8a2eea50f0dc4e1885f6a634ffee5659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca6fddfdcb04539a7503b9b83d107e9" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 34183 } } peers { permanent_uuid: "cab0258603694a59b6bb8466b626d11e" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 44973 } } peers { permanent_uuid: "8a2eea50f0dc4e1885f6a634ffee5659" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45749 } }
I20260929 20:21:11.001444 13603 raft_consensus.cc:374] T 583ba459d3c9497285b0b1cc0eb82595 P 8a2eea50f0dc4e1885f6a634ffee5659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:11.001566 13603 raft_consensus.cc:729] T 583ba459d3c9497285b0b1cc0eb82595 P 8a2eea50f0dc4e1885f6a634ffee5659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2eea50f0dc4e1885f6a634ffee5659, State: Initialized, Role: FOLLOWER
I20260929 20:21:11.002148 13603 consensus_queue.cc:260] T 583ba459d3c9497285b0b1cc0eb82595 P 8a2eea50f0dc4e1885f6a634ffee5659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca6fddfdcb04539a7503b9b83d107e9" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 34183 } } peers { permanent_uuid: "cab0258603694a59b6bb8466b626d11e" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 44973 } } peers { permanent_uuid: "8a2eea50f0dc4e1885f6a634ffee5659" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45749 } }
I20260929 20:21:11.003093 13581 heartbeater.cc:503] Master 127.27.185.126:37963 was elected leader, sending a full tablet report...
I20260929 20:21:11.003865 13603 ts_tablet_manager.cc:1428] T 583ba459d3c9497285b0b1cc0eb82595 P 8a2eea50f0dc4e1885f6a634ffee5659: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
W20260929 20:21:11.048015 13314 tablet.cc:2369] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:21:11.075717 13448 tablet.cc:2369] T 583ba459d3c9497285b0b1cc0eb82595 P 9ca6fddfdcb04539a7503b9b83d107e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:21:11.104133 13582 tablet.cc:2369] T 583ba459d3c9497285b0b1cc0eb82595 P 8a2eea50f0dc4e1885f6a634ffee5659: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:21:11.321280 13607 raft_consensus.cc:482] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:21:11.321662 13607 raft_consensus.cc:504] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca6fddfdcb04539a7503b9b83d107e9" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 34183 } } peers { permanent_uuid: "cab0258603694a59b6bb8466b626d11e" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 44973 } } peers { permanent_uuid: "8a2eea50f0dc4e1885f6a634ffee5659" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45749 } }
I20260929 20:21:11.323939 13607 leader_election.cc:288] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ca6fddfdcb04539a7503b9b83d107e9 (127.27.185.66:34183), 8a2eea50f0dc4e1885f6a634ffee5659 (127.27.185.67:45749)
I20260929 20:21:11.332000 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:11.324685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:57563 (local address 127.27.185.66:34183)
0929 20:21:11.324868 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:11.324874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:11.325378 (+ 504us) server_negotiation.cc:407] Connection header received
0929 20:21:11.325674 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:11.325684 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:11.325769 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:11.325840 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:21:11.326624 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:11.327770 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:11.329787 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:11.330927 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:11.331078 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:11.331153 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:11.331574 (+ 421us) server_negotiation.cc:299] Negotiation successful
0929 20:21:11.331729 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:21:11.332101 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:11.324352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:34183 (local address 127.27.185.65:57563)
0929 20:21:11.325249 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:11.325269 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:11.325470 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:11.326063 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:11.326070 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:11.326096 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:21:11.326464 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:11.326477 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:11.327912 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:11.327918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:11.329577 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:11.329597 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:11.331133 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:11.331139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:11.331316 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:11.331352 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 20:21:11.331512 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 20:21:11.331655 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":83,"thread_start_us":515,"threads_started":1}
I20260929 20:21:11.332858 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:11.324423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:48083 (local address 127.27.185.67:45749)
0929 20:21:11.324653 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:11.324663 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:11.326003 (+ 1340us) server_negotiation.cc:407] Connection header received
0929 20:21:11.326169 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:11.326175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:11.326250 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:11.326356 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:21:11.327046 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:11.327811 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:11.330420 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:11.331632 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:11.331774 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:11.331844 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:11.332454 (+ 610us) server_negotiation.cc:299] Negotiation successful
0929 20:21:11.332611 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 20:21:11.333526 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:11.325166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:45749 (local address 127.27.185.65:48083)
0929 20:21:11.325882 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:11.325901 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:11.326036 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:11.326392 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:11.326399 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:11.326431 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:21:11.326855 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:11.326868 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:11.327966 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:11.327975 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:11.330255 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:11.330269 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:11.332010 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:11.332020 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:11.332259 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:11.332295 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 20:21:11.333102 (+ 807us) client_negotiation.cc:240] Negotiation successful
0929 20:21:11.333258 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":41,"thread_start_us":395,"threads_started":1}
I20260929 20:21:11.333336 13402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "583ba459d3c9497285b0b1cc0eb82595" candidate_uuid: "cab0258603694a59b6bb8466b626d11e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca6fddfdcb04539a7503b9b83d107e9" is_pre_election: true
I20260929 20:21:11.333894 13402 raft_consensus.cc:2454] T 583ba459d3c9497285b0b1cc0eb82595 P 9ca6fddfdcb04539a7503b9b83d107e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab0258603694a59b6bb8466b626d11e in term 0.
I20260929 20:21:11.334549 13536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "583ba459d3c9497285b0b1cc0eb82595" candidate_uuid: "cab0258603694a59b6bb8466b626d11e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2eea50f0dc4e1885f6a634ffee5659" is_pre_election: true
I20260929 20:21:11.334633 13202 leader_election.cc:302] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ca6fddfdcb04539a7503b9b83d107e9, cab0258603694a59b6bb8466b626d11e; no voters:
I20260929 20:21:11.335032 13536 raft_consensus.cc:2454] T 583ba459d3c9497285b0b1cc0eb82595 P 8a2eea50f0dc4e1885f6a634ffee5659 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab0258603694a59b6bb8466b626d11e in term 0.
I20260929 20:21:11.335075 13607 raft_consensus.cc:2781] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:21:11.335223 13607 raft_consensus.cc:482] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:21:11.335341 13607 raft_consensus.cc:3037] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:11.337594 13607 raft_consensus.cc:504] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca6fddfdcb04539a7503b9b83d107e9" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 34183 } } peers { permanent_uuid: "cab0258603694a59b6bb8466b626d11e" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 44973 } } peers { permanent_uuid: "8a2eea50f0dc4e1885f6a634ffee5659" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45749 } }
I20260929 20:21:11.338213 13607 leader_election.cc:288] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e [CANDIDATE]: Term 1 election: Requested vote from peers 9ca6fddfdcb04539a7503b9b83d107e9 (127.27.185.66:34183), 8a2eea50f0dc4e1885f6a634ffee5659 (127.27.185.67:45749)
I20260929 20:21:11.338634 13402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "583ba459d3c9497285b0b1cc0eb82595" candidate_uuid: "cab0258603694a59b6bb8466b626d11e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca6fddfdcb04539a7503b9b83d107e9"
I20260929 20:21:11.338721 13536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "583ba459d3c9497285b0b1cc0eb82595" candidate_uuid: "cab0258603694a59b6bb8466b626d11e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2eea50f0dc4e1885f6a634ffee5659"
I20260929 20:21:11.338855 13402 raft_consensus.cc:3037] T 583ba459d3c9497285b0b1cc0eb82595 P 9ca6fddfdcb04539a7503b9b83d107e9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:11.338917 13536 raft_consensus.cc:3037] T 583ba459d3c9497285b0b1cc0eb82595 P 8a2eea50f0dc4e1885f6a634ffee5659 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:11.342131 13536 raft_consensus.cc:2454] T 583ba459d3c9497285b0b1cc0eb82595 P 8a2eea50f0dc4e1885f6a634ffee5659 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab0258603694a59b6bb8466b626d11e in term 1.
I20260929 20:21:11.342136 13402 raft_consensus.cc:2454] T 583ba459d3c9497285b0b1cc0eb82595 P 9ca6fddfdcb04539a7503b9b83d107e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab0258603694a59b6bb8466b626d11e in term 1.
I20260929 20:21:11.342549 13202 leader_election.cc:302] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ca6fddfdcb04539a7503b9b83d107e9, cab0258603694a59b6bb8466b626d11e; no voters:
I20260929 20:21:11.342851 13607 raft_consensus.cc:2781] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:21:11.343284 13607 raft_consensus.cc:686] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e [term 1 LEADER]: Becoming Leader. State: Replica: cab0258603694a59b6bb8466b626d11e, State: Running, Role: LEADER
I20260929 20:21:11.343779 13607 consensus_queue.cc:237] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca6fddfdcb04539a7503b9b83d107e9" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 34183 } } peers { permanent_uuid: "cab0258603694a59b6bb8466b626d11e" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 44973 } } peers { permanent_uuid: "8a2eea50f0dc4e1885f6a634ffee5659" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45749 } }
I20260929 20:21:11.348081 13124 catalog_manager.cc:5495] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e reported cstate change: term changed from 0 to 1, leader changed from <none> to cab0258603694a59b6bb8466b626d11e (127.27.185.65). New cstate: current_term: 1 leader_uuid: "cab0258603694a59b6bb8466b626d11e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca6fddfdcb04539a7503b9b83d107e9" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 34183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cab0258603694a59b6bb8466b626d11e" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 44973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a2eea50f0dc4e1885f6a634ffee5659" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45749 } health_report { overall_health: UNKNOWN } } }
I20260929 20:21:11.395360 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:11.389786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37614 (local address 127.27.185.65:44973)
0929 20:21:11.390011 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:11.390019 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:11.390097 (+ 78us) server_negotiation.cc:407] Connection header received
0929 20:21:11.390334 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:11.390338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:11.390415 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:11.390580 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:21:11.390952 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:11.391790 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:11.393318 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:11.393725 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:11.393827 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:11.394287 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:21:11.394804 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:21:11.394918 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:11.394994 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 20:21:11.395060 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:21:11.403931 13536 raft_consensus.cc:1261] T 583ba459d3c9497285b0b1cc0eb82595 P 8a2eea50f0dc4e1885f6a634ffee5659 [term 1 FOLLOWER]: Refusing update from remote peer cab0258603694a59b6bb8466b626d11e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:21:11.403945 13402 raft_consensus.cc:1261] T 583ba459d3c9497285b0b1cc0eb82595 P 9ca6fddfdcb04539a7503b9b83d107e9 [term 1 FOLLOWER]: Refusing update from remote peer cab0258603694a59b6bb8466b626d11e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:21:11.405057 13611 consensus_queue.cc:1035] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2eea50f0dc4e1885f6a634ffee5659" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:21:11.405488 13607 consensus_queue.cc:1035] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ca6fddfdcb04539a7503b9b83d107e9" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 34183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:21:11.433674 13123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:21:11.433985 13123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:21:11.437610 13123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc13acdf37b54e57b0b8970532a976bb: Sending DeleteTablet for 3 replicas of tablet 583ba459d3c9497285b0b1cc0eb82595
I20260929 20:21:11.438930 13516 tablet_service.cc:1514] Processing DeleteTablet for tablet 583ba459d3c9497285b0b1cc0eb82595 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:21:11 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:60516
I20260929 20:21:11.438930 13382 tablet_service.cc:1514] Processing DeleteTablet for tablet 583ba459d3c9497285b0b1cc0eb82595 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:21:11 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:58734
I20260929 20:21:11.440073 13248 tablet_service.cc:1514] Processing DeleteTablet for tablet 583ba459d3c9497285b0b1cc0eb82595 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:21:11 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:37600
I20260929 20:21:11.440385 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 13185
I20260929 20:21:11.440431 13624 tablet_replica.cc:330] T 583ba459d3c9497285b0b1cc0eb82595 P 8a2eea50f0dc4e1885f6a634ffee5659: stopping tablet replica
I20260929 20:21:11.440889 13624 raft_consensus.cc:2229] T 583ba459d3c9497285b0b1cc0eb82595 P 8a2eea50f0dc4e1885f6a634ffee5659 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:21:11.441445 13624 raft_consensus.cc:2258] T 583ba459d3c9497285b0b1cc0eb82595 P 8a2eea50f0dc4e1885f6a634ffee5659 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:21:11.442274 13625 tablet_replica.cc:330] T 583ba459d3c9497285b0b1cc0eb82595 P 9ca6fddfdcb04539a7503b9b83d107e9: stopping tablet replica
I20260929 20:21:11.442729 13625 raft_consensus.cc:2229] T 583ba459d3c9497285b0b1cc0eb82595 P 9ca6fddfdcb04539a7503b9b83d107e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:21:11.443244 13625 raft_consensus.cc:2258] T 583ba459d3c9497285b0b1cc0eb82595 P 9ca6fddfdcb04539a7503b9b83d107e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:21:11.444953 13624 ts_tablet_manager.cc:1905] T 583ba459d3c9497285b0b1cc0eb82595 P 8a2eea50f0dc4e1885f6a634ffee5659: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:21:11.450181 13625 ts_tablet_manager.cc:1905] T 583ba459d3c9497285b0b1cc0eb82595 P 9ca6fddfdcb04539a7503b9b83d107e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:21:11.450558 13626 tablet_replica.cc:330] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e: stopping tablet replica
I20260929 20:21:11.451076 13626 raft_consensus.cc:2229] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:21:11.451872 13626 raft_consensus.cc:2258] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:21:11.454120 13626 ts_tablet_manager.cc:1905] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:21:11.456466 13624 ts_tablet_manager.cc:1918] T 583ba459d3c9497285b0b1cc0eb82595 P 8a2eea50f0dc4e1885f6a634ffee5659: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:21:11.456729 13624 log.cc:1196] T 583ba459d3c9497285b0b1cc0eb82595 P 8a2eea50f0dc4e1885f6a634ffee5659: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-2/wal/wals/583ba459d3c9497285b0b1cc0eb82595
I20260929 20:21:11.457377 13624 ts_tablet_manager.cc:1939] T 583ba459d3c9497285b0b1cc0eb82595 P 8a2eea50f0dc4e1885f6a634ffee5659: Deleting consensus metadata
I20260929 20:21:11.459707 13108 catalog_manager.cc:4841] TS 8a2eea50f0dc4e1885f6a634ffee5659 (127.27.185.67:45749): tablet 583ba459d3c9497285b0b1cc0eb82595 (table test-table [id=5e39d755f93a4a56bd0c9465d6334101]) successfully deleted
I20260929 20:21:11.461743 13626 ts_tablet_manager.cc:1918] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:21:11.462018 13626 log.cc:1196] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-0/wal/wals/583ba459d3c9497285b0b1cc0eb82595
I20260929 20:21:11.462705 13626 ts_tablet_manager.cc:1939] T 583ba459d3c9497285b0b1cc0eb82595 P cab0258603694a59b6bb8466b626d11e: Deleting consensus metadata
I20260929 20:21:11.463543 13625 ts_tablet_manager.cc:1918] T 583ba459d3c9497285b0b1cc0eb82595 P 9ca6fddfdcb04539a7503b9b83d107e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:21:11.463766 13625 log.cc:1196] T 583ba459d3c9497285b0b1cc0eb82595 P 9ca6fddfdcb04539a7503b9b83d107e9: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790712966917858-28389-0/minicluster-data/ts-1/wal/wals/583ba459d3c9497285b0b1cc0eb82595
I20260929 20:21:11.464360 13625 ts_tablet_manager.cc:1939] T 583ba459d3c9497285b0b1cc0eb82595 P 9ca6fddfdcb04539a7503b9b83d107e9: Deleting consensus metadata
I20260929 20:21:11.464835 13108 catalog_manager.cc:4841] TS cab0258603694a59b6bb8466b626d11e (127.27.185.65:44973): tablet 583ba459d3c9497285b0b1cc0eb82595 (table test-table [id=5e39d755f93a4a56bd0c9465d6334101]) successfully deleted
I20260929 20:21:11.466153 13109 catalog_manager.cc:4841] TS 9ca6fddfdcb04539a7503b9b83d107e9 (127.27.185.66:34183): tablet 583ba459d3c9497285b0b1cc0eb82595 (table test-table [id=5e39d755f93a4a56bd0c9465d6334101]) successfully deleted
Sep 29 20:21:11 dist-test-slave-4cx3 krb5kdc[13079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.65@KRBTEST.COM
I20260929 20:21:11.473943 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:11.442421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37624 (local address 127.27.185.65:44973)
0929 20:21:11.442604 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:11.442613 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:11.451921 (+ 9308us) server_negotiation.cc:407] Connection header received
0929 20:21:11.452240 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:11.452246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:11.452333 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:11.452436 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:11.457875 (+ 5439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:11.459180 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:11.461231 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:11.461985 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:11.462120 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:11.467823 (+ 5703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:11.467835 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:11.467852 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:11.467901 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:11.470902 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:11.471836 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:11.471846 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:11.471848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:11.471989 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:11.472418 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:11.472421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:11.472423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:11.472769 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:11.473007 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:11.473408 (+ 401us) server_negotiation.cc:299] Negotiation successful
0929 20:21:11.473605 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:21:11.474678 13308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:11.574064 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 13185
I20260929 20:21:11.592978 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 13319
Sep 29 20:21:11 dist-test-slave-4cx3 krb5kdc[13079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.66@KRBTEST.COM
I20260929 20:21:11.608072 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:11.593793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58740 (local address 127.27.185.66:34183)
0929 20:21:11.593995 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:11.594000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:11.594020 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:21:11.594132 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:11.594136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:11.594198 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:11.594284 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:11.595597 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:11.596311 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:11.597646 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:11.598039 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:11.598144 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:11.602583 (+ 4439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:11.602600 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:11.602615 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:11.602659 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:11.605504 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:11.606187 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:11.606194 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:11.606196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:11.606325 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:11.606624 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:11.606627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:11.606629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:11.606984 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:11.607202 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:11.607644 (+ 442us) server_negotiation.cc:299] Negotiation successful
0929 20:21:11.607809 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 20:21:11.608742 13442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:11.700361 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 13319
I20260929 20:21:11.717751 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 13453
Sep 29 20:21:11 dist-test-slave-4cx3 krb5kdc[13079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.67@KRBTEST.COM
I20260929 20:21:11.733304 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:11.718631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60522 (local address 127.27.185.67:45749)
0929 20:21:11.718818 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:11.718827 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:11.718849 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:21:11.719057 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:11.719061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:11.719127 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:11.719231 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:11.720676 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:11.721392 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:11.722809 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:11.723197 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:11.723333 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:11.727651 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:11.727666 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:11.727682 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:11.727716 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:11.730320 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:11.731064 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:11.731072 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:11.731075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:11.731210 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:11.731666 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:11.731671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:11.731674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:11.732058 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:11.732357 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:11.732720 (+ 363us) server_negotiation.cc:299] Negotiation successful
0929 20:21:11.732943 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:21:11.733948 13576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:11.822101 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 13453
I20260929 20:21:11.839047 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 13092
I20260929 20:21:11.840234 13154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:11.950414 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 13092
2026-09-29T20:21:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4627 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T20:21:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:21:12Z Disabled control of system clock
I20260929 20:21:12.028566 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:36107
--webserver_interface=127.27.185.126
--webserver_port=0
--builtin_ntp_servers=127.27.185.84:33567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:36107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 20:21:12.366888 13635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:12.367200 13635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:12.367352 13635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:12.374708 13635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:21:12.374821 13635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:12.374910 13635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:12.375018 13635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:21:12.375087 13635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:21:12.384586 13635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:33567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:36107
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:36107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:12.385985 13635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:12.387895 13635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:12.399845 13640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:12.399956 13641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:12.399925 13643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:12.401206 13635 server_base.cc:1034] running on GCE node
I20260929 20:21:12.401973 13635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:12.403357 13635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:12.404613 13635 hybrid_clock.cc:648] HybridClock initialized: now 1790713272404570 us; error 36 us; skew 500 ppm
I20260929 20:21:12.404968 13635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:21:12.407258 13635 webserver.cc:460] Webserver started at http://127.27.185.126:45237/ using document root <none> and password file <none>
I20260929 20:21:12.407967 13635 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:12.408088 13635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:12.408411 13635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:12.411237 13635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "654d93e0f288457d943e836fb1a25476"
format_stamp: "Formatted at 2026-09-29 20:21:12 on dist-test-slave-4cx3"
server_key: "ef636ad68c3942edd5e9481475aa01e8"
server_key_iv: "1cca9b6406a8a9fcb99807bda6bc660b"
server_key_version: "encryptionkey@0"
I20260929 20:21:12.412107 13635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "654d93e0f288457d943e836fb1a25476"
format_stamp: "Formatted at 2026-09-29 20:21:12 on dist-test-slave-4cx3"
server_key: "ef636ad68c3942edd5e9481475aa01e8"
server_key_iv: "1cca9b6406a8a9fcb99807bda6bc660b"
server_key_version: "encryptionkey@0"
I20260929 20:21:12.417122 13635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:21:12.420500 13649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:12.421787 13635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:21:12.421973 13635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "654d93e0f288457d943e836fb1a25476"
format_stamp: "Formatted at 2026-09-29 20:21:12 on dist-test-slave-4cx3"
server_key: "ef636ad68c3942edd5e9481475aa01e8"
server_key_iv: "1cca9b6406a8a9fcb99807bda6bc660b"
server_key_version: "encryptionkey@0"
I20260929 20:21:12.422194 13635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:12.457721 13635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:12.458660 13635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:12.458954 13635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:12.477782 13635 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:36107
I20260929 20:21:12.477793 13700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:36107 every 8 connection(s)
I20260929 20:21:12.479148 13635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:21:12.482942 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 13635
I20260929 20:21:12.483103 13701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:12.483139 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:21:12.483735 28389 external_mini_cluster.cc:1428] Setting key c54940fca61368c7ffc3623e5f802bc2
I20260929 20:21:12.496621 13701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 654d93e0f288457d943e836fb1a25476: Bootstrap starting.
I20260929 20:21:12.498195 13704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:12.485690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35598 (local address 127.27.185.126:36107)
0929 20:21:12.486745 (+ 1055us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:12.486762 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:12.486805 (+ 43us) server_negotiation.cc:407] Connection header received
0929 20:21:12.489133 (+ 2328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:12.489157 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:12.489648 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:12.489983 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:12.490563 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:12.491883 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:12.493898 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:12.494588 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:12.494711 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:12.495969 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:12.495977 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:12.495991 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:12.496519 (+ 528us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:12.496593 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:12.496617 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:12.496702 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:12.496952 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 20:21:12.497285 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":417,"threads_started":1}
I20260929 20:21:12.500025 13701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 654d93e0f288457d943e836fb1a25476: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:12.501130 13701 log.cc:824] T 00000000000000000000000000000000 P 654d93e0f288457d943e836fb1a25476: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:12.503937 13701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 654d93e0f288457d943e836fb1a25476: No bootstrap required, opened a new log
I20260929 20:21:12.513237 13701 raft_consensus.cc:348] T 00000000000000000000000000000000 P 654d93e0f288457d943e836fb1a25476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654d93e0f288457d943e836fb1a25476" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36107 } }
I20260929 20:21:12.513761 13701 raft_consensus.cc:374] T 00000000000000000000000000000000 P 654d93e0f288457d943e836fb1a25476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:12.513877 13701 raft_consensus.cc:729] T 00000000000000000000000000000000 P 654d93e0f288457d943e836fb1a25476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 654d93e0f288457d943e836fb1a25476, State: Initialized, Role: FOLLOWER
I20260929 20:21:12.514477 13701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 654d93e0f288457d943e836fb1a25476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654d93e0f288457d943e836fb1a25476" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36107 } }
I20260929 20:21:12.514770 13701 raft_consensus.cc:388] T 00000000000000000000000000000000 P 654d93e0f288457d943e836fb1a25476 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:21:12.514940 13701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 654d93e0f288457d943e836fb1a25476 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:21:12.515110 13701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 654d93e0f288457d943e836fb1a25476 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:12.517508 13701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 654d93e0f288457d943e836fb1a25476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654d93e0f288457d943e836fb1a25476" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36107 } }
I20260929 20:21:12.517978 13701 leader_election.cc:302] T 00000000000000000000000000000000 P 654d93e0f288457d943e836fb1a25476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 654d93e0f288457d943e836fb1a25476; no voters:
I20260929 20:21:12.518512 13701 leader_election.cc:288] T 00000000000000000000000000000000 P 654d93e0f288457d943e836fb1a25476 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:21:12.518677 13706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 654d93e0f288457d943e836fb1a25476 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:21:12.519662 13706 raft_consensus.cc:686] T 00000000000000000000000000000000 P 654d93e0f288457d943e836fb1a25476 [term 1 LEADER]: Becoming Leader. State: Replica: 654d93e0f288457d943e836fb1a25476, State: Running, Role: LEADER
I20260929 20:21:12.520164 13706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 654d93e0f288457d943e836fb1a25476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654d93e0f288457d943e836fb1a25476" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36107 } }
I20260929 20:21:12.520588 13701 sys_catalog.cc:564] T 00000000000000000000000000000000 P 654d93e0f288457d943e836fb1a25476 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:21:12.523176 13707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 654d93e0f288457d943e836fb1a25476 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "654d93e0f288457d943e836fb1a25476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654d93e0f288457d943e836fb1a25476" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36107 } } }
I20260929 20:21:12.523263 13708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 654d93e0f288457d943e836fb1a25476 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 654d93e0f288457d943e836fb1a25476. Latest consensus state: current_term: 1 leader_uuid: "654d93e0f288457d943e836fb1a25476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654d93e0f288457d943e836fb1a25476" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 36107 } } }
I20260929 20:21:12.523604 13707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 654d93e0f288457d943e836fb1a25476 [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:12.523680 13708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 654d93e0f288457d943e836fb1a25476 [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:12.525283 13712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:21:12.530522 13712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:21:12.539832 13712 catalog_manager.cc:1347] Generated new cluster ID: 52a790147efe48aca29de4dbad426131
I20260929 20:21:12.539937 13712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:21:12.561201 13712 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:21:12.562501 13712 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:21:12.575841 13712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 654d93e0f288457d943e836fb1a25476: Generated new TSK 0
I20260929 20:21:12.576752 13712 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:21:12.588568 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:0
--local_ip_for_outbound_sockets=127.27.185.65
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:36107
--builtin_ntp_servers=127.27.185.84:33567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 20:21:12.944015 13725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:12.944288 13725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:12.944373 13725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:12.951771 13725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:12.951871 13725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:12.951974 13725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:21:12.961259 13725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:33567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:36107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:12.962544 13725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:12.965147 13725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:12.979909 13730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:12.980049 13733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:12.980086 13731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:12.981113 13725 server_base.cc:1034] running on GCE node
I20260929 20:21:12.981797 13725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:12.982829 13725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:12.984062 13725 hybrid_clock.cc:648] HybridClock initialized: now 1790713272984004 us; error 48 us; skew 500 ppm
I20260929 20:21:12.984397 13725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:21:12.986753 13725 webserver.cc:460] Webserver started at http://127.27.185.65:40997/ using document root <none> and password file <none>
I20260929 20:21:12.987582 13725 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:12.987715 13725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:12.988044 13725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:12.991137 13725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-0/data/instance:
uuid: "ac5052f67e2d438e8399b8ee2e211075"
format_stamp: "Formatted at 2026-09-29 20:21:12 on dist-test-slave-4cx3"
server_key: "014efd4ceeb955258d854eeca0e5f926"
server_key_iv: "915e387f8f9b44bb8ec8acb1d217d609"
server_key_version: "encryptionkey@0"
I20260929 20:21:12.992043 13725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance:
uuid: "ac5052f67e2d438e8399b8ee2e211075"
format_stamp: "Formatted at 2026-09-29 20:21:12 on dist-test-slave-4cx3"
server_key: "014efd4ceeb955258d854eeca0e5f926"
server_key_iv: "915e387f8f9b44bb8ec8acb1d217d609"
server_key_version: "encryptionkey@0"
I20260929 20:21:12.997380 13725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:21:13.001523 13739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:13.003101 13725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:21:13.003286 13725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "ac5052f67e2d438e8399b8ee2e211075"
format_stamp: "Formatted at 2026-09-29 20:21:12 on dist-test-slave-4cx3"
server_key: "014efd4ceeb955258d854eeca0e5f926"
server_key_iv: "915e387f8f9b44bb8ec8acb1d217d609"
server_key_version: "encryptionkey@0"
I20260929 20:21:13.003595 13725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:13.039038 13725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:13.040027 13725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:13.040349 13725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:13.041378 13725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:21:13.042930 13725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:21:13.043051 13725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:13.043179 13725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:21:13.043264 13725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:13.076656 13725 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:45193
I20260929 20:21:13.076749 13851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:45193 every 8 connection(s)
I20260929 20:21:13.078114 13725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:21:13.082765 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 13725
I20260929 20:21:13.082932 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance
I20260929 20:21:13.083350 28389 external_mini_cluster.cc:1428] Setting key 2b64d766c4937f0fa7af64c68acfd30c
I20260929 20:21:13.086458 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:0
--local_ip_for_outbound_sockets=127.27.185.66
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:36107
--builtin_ntp_servers=127.27.185.84:33567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 20:21:13.091732 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:13.080762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:43483 (local address 127.27.185.126:36107)
0929 20:21:13.081152 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:13.081157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:13.082418 (+ 1261us) server_negotiation.cc:407] Connection header received
0929 20:21:13.084409 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:13.084414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:13.084493 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:13.084607 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:13.085780 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:13.086852 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:13.088500 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:13.089199 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:13.089350 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:13.090681 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:13.090689 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:13.090697 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:13.090724 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:13.090781 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:13.090800 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:13.090893 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:13.091103 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 20:21:13.091219 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":287,"thread_start_us":192,"threads_started":1}
I20260929 20:21:13.092629 13855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:13.081387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36107 (local address 127.27.185.65:43483)
0929 20:21:13.082262 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:13.082298 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:13.084086 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:13.084813 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:13.084822 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:13.084915 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:21:13.085615 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:13.085633 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:13.087004 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:13.087008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:13.088357 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:13.088370 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:13.089410 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:13.089415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:13.089628 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:13.090340 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:21:13.090343 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:21:13.090384 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:21:13.090471 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:21:13.090901 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:21:13.090909 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:21:13.090915 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 20:21:13.091085 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 20:21:13.091459 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":248,"threads_started":1}
I20260929 20:21:13.094197 13852 heartbeater.cc:347] Connected to a master server at 127.27.185.126:36107
I20260929 20:21:13.094560 13852 heartbeater.cc:464] Registering TS with master...
I20260929 20:21:13.095547 13852 heartbeater.cc:511] Master 127.27.185.126:36107 requested a full tablet report, sending...
I20260929 20:21:13.097654 13666 ts_manager.cc:194] Registered new tserver with Master: ac5052f67e2d438e8399b8ee2e211075 (127.27.185.65:45193)
I20260929 20:21:13.099123 13666 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.185.65:43483
W20260929 20:21:13.423208 13856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:13.423574 13856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:13.423727 13856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:13.431308 13856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:13.431468 13856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:13.431648 13856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:21:13.441370 13856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:33567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:36107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:13.442739 13856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:13.444629 13856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:13.458014 13861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:13.457952 13864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:13.458063 13862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:13.458581 13856 server_base.cc:1034] running on GCE node
I20260929 20:21:13.459419 13856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:13.460361 13856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:13.461591 13856 hybrid_clock.cc:648] HybridClock initialized: now 1790713273461520 us; error 33 us; skew 500 ppm
I20260929 20:21:13.461925 13856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:21:13.464517 13856 webserver.cc:460] Webserver started at http://127.27.185.66:36233/ using document root <none> and password file <none>
I20260929 20:21:13.465272 13856 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:13.465385 13856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:13.465715 13856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:13.468673 13856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-1/data/instance:
uuid: "9fbb871491ad41d8b142074d92ae9e6b"
format_stamp: "Formatted at 2026-09-29 20:21:13 on dist-test-slave-4cx3"
server_key: "42ad75507d84419e65a665eac2bbcabc"
server_key_iv: "f825ca85044e5ccc55321d73b9b09525"
server_key_version: "encryptionkey@0"
I20260929 20:21:13.469497 13856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance:
uuid: "9fbb871491ad41d8b142074d92ae9e6b"
format_stamp: "Formatted at 2026-09-29 20:21:13 on dist-test-slave-4cx3"
server_key: "42ad75507d84419e65a665eac2bbcabc"
server_key_iv: "f825ca85044e5ccc55321d73b9b09525"
server_key_version: "encryptionkey@0"
I20260929 20:21:13.474887 13856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 20:21:13.478605 13870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:13.480093 13856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:21:13.480258 13856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "9fbb871491ad41d8b142074d92ae9e6b"
format_stamp: "Formatted at 2026-09-29 20:21:13 on dist-test-slave-4cx3"
server_key: "42ad75507d84419e65a665eac2bbcabc"
server_key_iv: "f825ca85044e5ccc55321d73b9b09525"
server_key_version: "encryptionkey@0"
I20260929 20:21:13.480480 13856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:13.497876 13856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:13.498757 13856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:13.499066 13856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:13.500103 13856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:21:13.501572 13856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:21:13.501683 13856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:13.501806 13856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:21:13.501890 13856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:13.538931 13856 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:40233
I20260929 20:21:13.539000 13982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:40233 every 8 connection(s)
I20260929 20:21:13.540884 13856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:21:13.543229 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 13856
I20260929 20:21:13.543444 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance
I20260929 20:21:13.544060 28389 external_mini_cluster.cc:1428] Setting key 68875f7a57ae6bb44f8c4fc0e891e096
I20260929 20:21:13.549360 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:0
--local_ip_for_outbound_sockets=127.27.185.67
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:36107
--builtin_ntp_servers=127.27.185.84:33567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 20:21:13.560024 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:13.544925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:56975 (local address 127.27.185.126:36107)
0929 20:21:13.545157 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:13.545163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:13.547351 (+ 2188us) server_negotiation.cc:407] Connection header received
0929 20:21:13.550617 (+ 3266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:13.550627 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:13.550736 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:13.550855 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:13.552567 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:13.553739 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:13.556182 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:13.556908 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:13.557069 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:13.558973 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:13.558978 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:13.558982 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:13.559005 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:13.559041 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:13.559056 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:13.559156 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:13.559506 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 20:21:13.559638 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:21:13.562582 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:13.545440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36107 (local address 127.27.185.66:56975)
0929 20:21:13.547145 (+ 1705us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:13.547214 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:13.550192 (+ 2978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:13.551175 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:13.551188 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:13.551339 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:21:13.552340 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:13.552376 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:13.553932 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:13.553937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:13.555996 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:13.556012 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:13.557137 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:13.557142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:13.557437 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:13.558481 (+ 1044us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:21:13.558485 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:21:13.558544 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:21:13.558669 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:21:13.559155 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:21:13.559170 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:21:13.559181 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 20:21:13.560383 (+ 1202us) client_negotiation.cc:240] Negotiation successful
0929 20:21:13.560918 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":993,"thread_start_us":726,"threads_started":1}
I20260929 20:21:13.564527 13983 heartbeater.cc:347] Connected to a master server at 127.27.185.126:36107
I20260929 20:21:13.564926 13983 heartbeater.cc:464] Registering TS with master...
I20260929 20:21:13.566009 13983 heartbeater.cc:511] Master 127.27.185.126:36107 requested a full tablet report, sending...
I20260929 20:21:13.568089 13666 ts_manager.cc:194] Registered new tserver with Master: 9fbb871491ad41d8b142074d92ae9e6b (127.27.185.66:40233)
I20260929 20:21:13.569118 13666 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.185.66:56975
W20260929 20:21:13.906778 13986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:13.907068 13986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:13.907155 13986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:13.914669 13986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:13.914773 13986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:13.914880 13986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:21:13.924173 13986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:33567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:36107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:13.925560 13986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:13.927598 13986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:13.941283 13991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:13.941319 13994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:13.941370 13992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:13.942179 13986 server_base.cc:1034] running on GCE node
I20260929 20:21:13.942740 13986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:13.943775 13986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:13.944999 13986 hybrid_clock.cc:648] HybridClock initialized: now 1790713273944955 us; error 54 us; skew 500 ppm
I20260929 20:21:13.945338 13986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:21:13.947788 13986 webserver.cc:460] Webserver started at http://127.27.185.67:42249/ using document root <none> and password file <none>
I20260929 20:21:13.948510 13986 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:13.948653 13986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:13.948968 13986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:13.952080 13986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-2/data/instance:
uuid: "43b0420fd0d8437d96c0594df6c1d57d"
format_stamp: "Formatted at 2026-09-29 20:21:13 on dist-test-slave-4cx3"
server_key: "bcb2a7d917b24669239ebafaa7457b59"
server_key_iv: "16cf79c7a51555dba1760efb5b303e63"
server_key_version: "encryptionkey@0"
I20260929 20:21:13.952965 13986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance:
uuid: "43b0420fd0d8437d96c0594df6c1d57d"
format_stamp: "Formatted at 2026-09-29 20:21:13 on dist-test-slave-4cx3"
server_key: "bcb2a7d917b24669239ebafaa7457b59"
server_key_iv: "16cf79c7a51555dba1760efb5b303e63"
server_key_version: "encryptionkey@0"
I20260929 20:21:13.958151 13986 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:21:13.961750 14000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:13.963152 13986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 20:21:13.963325 13986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "43b0420fd0d8437d96c0594df6c1d57d"
format_stamp: "Formatted at 2026-09-29 20:21:13 on dist-test-slave-4cx3"
server_key: "bcb2a7d917b24669239ebafaa7457b59"
server_key_iv: "16cf79c7a51555dba1760efb5b303e63"
server_key_version: "encryptionkey@0"
I20260929 20:21:13.963639 13986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:13.990084 13986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:13.990988 13986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:13.991313 13986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:13.992429 13986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:21:13.994076 13986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:21:13.994295 13986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:13.994454 13986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:21:13.994549 13986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:14.028798 13986 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:41839
I20260929 20:21:14.028865 14112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:41839 every 8 connection(s)
I20260929 20:21:14.030200 13986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:21:14.034016 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 13986
I20260929 20:21:14.034210 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance
I20260929 20:21:14.034729 28389 external_mini_cluster.cc:1428] Setting key 96988df33d986c4309b490d08d6f5173
I20260929 20:21:14.043064 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:14.032743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:55731 (local address 127.27.185.126:36107)
0929 20:21:14.032972 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:14.032978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:14.034470 (+ 1492us) server_negotiation.cc:407] Connection header received
0929 20:21:14.036711 (+ 2241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:14.036717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:14.036811 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:14.036921 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:14.038096 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.038852 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.040453 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.040859 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.041017 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:14.042251 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:14.042256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:14.042259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:14.042276 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:14.042307 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:14.042318 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:14.042415 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:14.042626 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 20:21:14.042722 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 20:21:14.044085 14115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:14.033294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:36107 (local address 127.27.185.67:55731)
0929 20:21:14.034240 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:14.034310 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:14.036430 (+ 2120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:14.037157 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:14.037165 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:14.037262 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:21:14.037941 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:14.037958 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.039001 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.039015 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:14.040281 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:14.040293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.041080 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.041089 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:14.041316 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:14.041943 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:21:14.041946 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:21:14.041984 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:21:14.042058 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:21:14.042415 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:21:14.042428 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:21:14.042437 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 20:21:14.042614 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 20:21:14.042992 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":272,"threads_started":1}
I20260929 20:21:14.045421 14113 heartbeater.cc:347] Connected to a master server at 127.27.185.126:36107
I20260929 20:21:14.045732 14113 heartbeater.cc:464] Registering TS with master...
I20260929 20:21:14.046484 14113 heartbeater.cc:511] Master 127.27.185.126:36107 requested a full tablet report, sending...
I20260929 20:21:14.047974 13666 ts_manager.cc:194] Registered new tserver with Master: 43b0420fd0d8437d96c0594df6c1d57d (127.27.185.67:41839)
I20260929 20:21:14.048918 13666 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.185.67:55731
I20260929 20:21:14.049953 28389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:21:14.060118 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:14.054581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35610 (local address 127.27.185.126:36107)
0929 20:21:14.054739 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:14.054744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:14.055170 (+ 426us) server_negotiation.cc:407] Connection header received
0929 20:21:14.055340 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:14.055344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:14.055465 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:14.055549 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:14.056007 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.056694 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.058075 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.058479 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.058579 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:14.059566 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:14.059571 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:14.059574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:14.059592 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:14.059622 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:14.059634 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:14.059705 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:14.059812 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 20:21:14.059883 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 20:21:14.063757 13666 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:21:14.070662 13666 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:21:14.102015 13852 heartbeater.cc:503] Master 127.27.185.126:36107 was elected leader, sending a full tablet report...
I20260929 20:21:14.106765 14128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:14.083207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:40233 (local address 127.0.0.1:57100)
0929 20:21:14.093630 (+ 10423us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:14.093686 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:14.093920 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:14.099272 (+ 5352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:14.099277 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:14.099301 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:21:14.099875 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:14.099886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.101449 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.101456 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:14.104013 (+ 2557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:14.104026 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.105759 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.105765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:14.105955 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:14.106027 (+ 72us) client_negotiation.cc:769] Sending connection context
0929 20:21:14.106160 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 20:21:14.106354 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10136,"mutex_wait_us":39,"thread_start_us":14805,"threads_started":1}
I20260929 20:21:14.107623 14130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:14.084198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52502 (local address 127.27.185.65:45193)
0929 20:21:14.096242 (+ 12044us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:14.096256 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:14.098597 (+ 2341us) server_negotiation.cc:407] Connection header received
0929 20:21:14.098696 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:14.098700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:14.098856 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:14.099076 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:21:14.099823 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.101418 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.104694 (+ 3276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.105595 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.105749 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:14.106010 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:14.107016 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0929 20:21:14.107194 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11784,"thread_start_us":12062,"threads_started":1}
I20260929 20:21:14.107698 14126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:14.083236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57100 (local address 127.27.185.66:40233)
0929 20:21:14.098625 (+ 15389us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:14.098635 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:14.098667 (+ 32us) server_negotiation.cc:407] Connection header received
0929 20:21:14.098771 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:14.098781 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:14.099037 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:14.099475 (+ 438us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:21:14.100047 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.101278 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.104200 (+ 2922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.105557 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.105699 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:14.105955 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:14.106253 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 20:21:14.106436 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15228,"thread_start_us":15127,"threads_started":1}
I20260929 20:21:14.106882 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:14.084008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:45193 (local address 127.0.0.1:52502)
0929 20:21:14.098349 (+ 14341us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:14.098371 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:14.098475 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:14.099043 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:14.099048 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:14.099076 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:21:14.099646 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:14.099660 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.101602 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.101607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:14.104444 (+ 2837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:14.104456 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.106134 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.106141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:14.106316 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:14.106346 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 20:21:14.106415 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 20:21:14.106535 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14201,"mutex_wait_us":68,"thread_start_us":8793,"threads_started":1}
I20260929 20:21:14.108251 14129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:14.083796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52074 (local address 127.27.185.67:41839)
0929 20:21:14.097073 (+ 13277us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:14.097085 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:14.097738 (+ 653us) server_negotiation.cc:407] Connection header received
0929 20:21:14.097964 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:14.097975 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:14.098295 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:14.098533 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:21:14.099750 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.101429 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.104677 (+ 3248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.106078 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.106229 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:14.106473 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:14.107704 (+ 1231us) server_negotiation.cc:299] Negotiation successful
0929 20:21:14.107874 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13028,"thread_start_us":12902,"threads_started":1}
I20260929 20:21:14.108269 14127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:14.083222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:41839 (local address 127.0.0.1:52074)
0929 20:21:14.097569 (+ 14347us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:14.097595 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:14.097770 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:14.098872 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:14.098879 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:14.098909 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:21:14.099511 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:14.099530 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.101599 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.101607 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:14.104441 (+ 2834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:14.104456 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.106846 (+ 2390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.106853 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:14.107051 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:14.107082 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 20:21:14.107158 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 20:21:14.107925 (+ 767us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14148,"mutex_wait_us":383,"thread_start_us":13746,"threads_started":1}
I20260929 20:21:14.111631 13918 tablet_service.cc:1467] Processing CreateTablet for tablet 628d1a5fdb5a4f96b96c80fe11832d09 (DEFAULT_TABLE table=test-table [id=c73d46d69fe34f8cbc1e233cb1547f3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:21:14.111799 13787 tablet_service.cc:1467] Processing CreateTablet for tablet 628d1a5fdb5a4f96b96c80fe11832d09 (DEFAULT_TABLE table=test-table [id=c73d46d69fe34f8cbc1e233cb1547f3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:21:14.112872 13918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 628d1a5fdb5a4f96b96c80fe11832d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:14.112527 14048 tablet_service.cc:1467] Processing CreateTablet for tablet 628d1a5fdb5a4f96b96c80fe11832d09 (DEFAULT_TABLE table=test-table [id=c73d46d69fe34f8cbc1e233cb1547f3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:21:14.113157 13787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 628d1a5fdb5a4f96b96c80fe11832d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:14.113886 14048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 628d1a5fdb5a4f96b96c80fe11832d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:14.124598 14131 tablet_bootstrap.cc:492] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b: Bootstrap starting.
I20260929 20:21:14.128547 14131 tablet_bootstrap.cc:654] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:14.129465 14133 tablet_bootstrap.cc:492] T 628d1a5fdb5a4f96b96c80fe11832d09 P 43b0420fd0d8437d96c0594df6c1d57d: Bootstrap starting.
I20260929 20:21:14.129920 14131 log.cc:824] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:14.131245 14132 tablet_bootstrap.cc:492] T 628d1a5fdb5a4f96b96c80fe11832d09 P ac5052f67e2d438e8399b8ee2e211075: Bootstrap starting.
I20260929 20:21:14.134189 14133 tablet_bootstrap.cc:654] T 628d1a5fdb5a4f96b96c80fe11832d09 P 43b0420fd0d8437d96c0594df6c1d57d: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:14.134358 14132 tablet_bootstrap.cc:654] T 628d1a5fdb5a4f96b96c80fe11832d09 P ac5052f67e2d438e8399b8ee2e211075: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:14.136508 14132 log.cc:824] T 628d1a5fdb5a4f96b96c80fe11832d09 P ac5052f67e2d438e8399b8ee2e211075: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:14.137569 14133 log.cc:824] T 628d1a5fdb5a4f96b96c80fe11832d09 P 43b0420fd0d8437d96c0594df6c1d57d: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:14.139582 14132 tablet_bootstrap.cc:492] T 628d1a5fdb5a4f96b96c80fe11832d09 P ac5052f67e2d438e8399b8ee2e211075: No bootstrap required, opened a new log
I20260929 20:21:14.139858 14132 ts_tablet_manager.cc:1397] T 628d1a5fdb5a4f96b96c80fe11832d09 P ac5052f67e2d438e8399b8ee2e211075: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 20:21:14.140661 14131 tablet_bootstrap.cc:492] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b: No bootstrap required, opened a new log
I20260929 20:21:14.140957 14131 ts_tablet_manager.cc:1397] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.009s
I20260929 20:21:14.141176 14133 tablet_bootstrap.cc:492] T 628d1a5fdb5a4f96b96c80fe11832d09 P 43b0420fd0d8437d96c0594df6c1d57d: No bootstrap required, opened a new log
I20260929 20:21:14.141446 14133 ts_tablet_manager.cc:1397] T 628d1a5fdb5a4f96b96c80fe11832d09 P 43b0420fd0d8437d96c0594df6c1d57d: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260929 20:21:14.147097 14132 raft_consensus.cc:348] T 628d1a5fdb5a4f96b96c80fe11832d09 P ac5052f67e2d438e8399b8ee2e211075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbb871491ad41d8b142074d92ae9e6b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 40233 } } peers { permanent_uuid: "43b0420fd0d8437d96c0594df6c1d57d" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 41839 } } peers { permanent_uuid: "ac5052f67e2d438e8399b8ee2e211075" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 45193 } }
I20260929 20:21:14.147640 14132 raft_consensus.cc:374] T 628d1a5fdb5a4f96b96c80fe11832d09 P ac5052f67e2d438e8399b8ee2e211075 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:14.147746 14132 raft_consensus.cc:729] T 628d1a5fdb5a4f96b96c80fe11832d09 P ac5052f67e2d438e8399b8ee2e211075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac5052f67e2d438e8399b8ee2e211075, State: Initialized, Role: FOLLOWER
I20260929 20:21:14.148551 14132 consensus_queue.cc:260] T 628d1a5fdb5a4f96b96c80fe11832d09 P ac5052f67e2d438e8399b8ee2e211075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbb871491ad41d8b142074d92ae9e6b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 40233 } } peers { permanent_uuid: "43b0420fd0d8437d96c0594df6c1d57d" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 41839 } } peers { permanent_uuid: "ac5052f67e2d438e8399b8ee2e211075" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 45193 } }
I20260929 20:21:14.150436 14132 ts_tablet_manager.cc:1428] T 628d1a5fdb5a4f96b96c80fe11832d09 P ac5052f67e2d438e8399b8ee2e211075: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260929 20:21:14.151938 14133 raft_consensus.cc:348] T 628d1a5fdb5a4f96b96c80fe11832d09 P 43b0420fd0d8437d96c0594df6c1d57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbb871491ad41d8b142074d92ae9e6b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 40233 } } peers { permanent_uuid: "43b0420fd0d8437d96c0594df6c1d57d" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 41839 } } peers { permanent_uuid: "ac5052f67e2d438e8399b8ee2e211075" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 45193 } }
I20260929 20:21:14.151855 14131 raft_consensus.cc:348] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbb871491ad41d8b142074d92ae9e6b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 40233 } } peers { permanent_uuid: "43b0420fd0d8437d96c0594df6c1d57d" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 41839 } } peers { permanent_uuid: "ac5052f67e2d438e8399b8ee2e211075" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 45193 } }
I20260929 20:21:14.152360 14131 raft_consensus.cc:374] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:14.152360 14133 raft_consensus.cc:374] T 628d1a5fdb5a4f96b96c80fe11832d09 P 43b0420fd0d8437d96c0594df6c1d57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:14.152470 14131 raft_consensus.cc:729] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fbb871491ad41d8b142074d92ae9e6b, State: Initialized, Role: FOLLOWER
I20260929 20:21:14.152472 14133 raft_consensus.cc:729] T 628d1a5fdb5a4f96b96c80fe11832d09 P 43b0420fd0d8437d96c0594df6c1d57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43b0420fd0d8437d96c0594df6c1d57d, State: Initialized, Role: FOLLOWER
I20260929 20:21:14.153110 14133 consensus_queue.cc:260] T 628d1a5fdb5a4f96b96c80fe11832d09 P 43b0420fd0d8437d96c0594df6c1d57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbb871491ad41d8b142074d92ae9e6b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 40233 } } peers { permanent_uuid: "43b0420fd0d8437d96c0594df6c1d57d" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 41839 } } peers { permanent_uuid: "ac5052f67e2d438e8399b8ee2e211075" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 45193 } }
I20260929 20:21:14.153110 14131 consensus_queue.cc:260] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbb871491ad41d8b142074d92ae9e6b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 40233 } } peers { permanent_uuid: "43b0420fd0d8437d96c0594df6c1d57d" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 41839 } } peers { permanent_uuid: "ac5052f67e2d438e8399b8ee2e211075" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 45193 } }
I20260929 20:21:14.154860 13983 heartbeater.cc:503] Master 127.27.185.126:36107 was elected leader, sending a full tablet report...
I20260929 20:21:14.155097 14131 ts_tablet_manager.cc:1428] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260929 20:21:14.155334 14133 ts_tablet_manager.cc:1428] T 628d1a5fdb5a4f96b96c80fe11832d09 P 43b0420fd0d8437d96c0594df6c1d57d: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260929 20:21:14.155608 14113 heartbeater.cc:503] Master 127.27.185.126:36107 was elected leader, sending a full tablet report...
I20260929 20:21:14.254688 14138 raft_consensus.cc:482] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:21:14.255020 14138 raft_consensus.cc:504] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbb871491ad41d8b142074d92ae9e6b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 40233 } } peers { permanent_uuid: "43b0420fd0d8437d96c0594df6c1d57d" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 41839 } } peers { permanent_uuid: "ac5052f67e2d438e8399b8ee2e211075" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 45193 } }
I20260929 20:21:14.256803 14138 leader_election.cc:288] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43b0420fd0d8437d96c0594df6c1d57d (127.27.185.67:41839), ac5052f67e2d438e8399b8ee2e211075 (127.27.185.65:45193)
I20260929 20:21:14.263793 14130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:14.257360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:41765 (local address 127.27.185.65:45193)
0929 20:21:14.257607 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:14.257616 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:14.258286 (+ 670us) server_negotiation.cc:407] Connection header received
0929 20:21:14.258460 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:14.258464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:14.258562 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:14.258660 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:21:14.259362 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.260077 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.262029 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.262825 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.262993 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:14.263084 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:14.263424 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 20:21:14.263552 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 20:21:14.263810 14129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:14.257123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:38599 (local address 127.27.185.67:41839)
0929 20:21:14.257363 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:14.257373 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:14.258135 (+ 762us) server_negotiation.cc:407] Connection header received
0929 20:21:14.258398 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:14.258405 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:14.258493 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:14.258562 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:21:14.259231 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.260035 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.262076 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.262835 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.263018 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:14.263084 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:14.263455 (+ 371us) server_negotiation.cc:299] Negotiation successful
0929 20:21:14.263583 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:21:14.263900 14140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:14.257048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:41839 (local address 127.27.185.66:38599)
0929 20:21:14.258001 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:14.258018 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:14.258209 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:14.258623 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:14.258629 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:14.258657 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:21:14.259084 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:14.259094 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.260171 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.260180 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:14.261871 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:14.261883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.263045 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.263052 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:14.263267 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:14.263290 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:21:14.263413 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 20:21:14.263545 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":176,"thread_start_us":556,"threads_started":1}
I20260929 20:21:14.263952 14141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:14.257132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:45193 (local address 127.27.185.66:41765)
0929 20:21:14.258176 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:14.258190 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:14.258327 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:14.258697 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:14.258702 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:14.258781 (+ 79us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:21:14.259188 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:14.259200 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.260236 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.260240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:14.261836 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:14.261848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.263036 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.263043 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:14.263203 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:14.263272 (+ 69us) client_negotiation.cc:769] Sending connection context
0929 20:21:14.263344 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 20:21:14.263513 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"mutex_wait_us":80,"spinlock_wait_cycles":32384,"thread_start_us":699,"threads_started":1}
I20260929 20:21:14.265045 14068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628d1a5fdb5a4f96b96c80fe11832d09" candidate_uuid: "9fbb871491ad41d8b142074d92ae9e6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43b0420fd0d8437d96c0594df6c1d57d" is_pre_election: true
I20260929 20:21:14.265030 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628d1a5fdb5a4f96b96c80fe11832d09" candidate_uuid: "9fbb871491ad41d8b142074d92ae9e6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac5052f67e2d438e8399b8ee2e211075" is_pre_election: true
I20260929 20:21:14.265556 14068 raft_consensus.cc:2454] T 628d1a5fdb5a4f96b96c80fe11832d09 P 43b0420fd0d8437d96c0594df6c1d57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fbb871491ad41d8b142074d92ae9e6b in term 0.
I20260929 20:21:14.265551 13807 raft_consensus.cc:2454] T 628d1a5fdb5a4f96b96c80fe11832d09 P ac5052f67e2d438e8399b8ee2e211075 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fbb871491ad41d8b142074d92ae9e6b in term 0.
I20260929 20:21:14.266211 13871 leader_election.cc:302] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 43b0420fd0d8437d96c0594df6c1d57d, 9fbb871491ad41d8b142074d92ae9e6b, ac5052f67e2d438e8399b8ee2e211075; no voters:
I20260929 20:21:14.266587 14138 raft_consensus.cc:2781] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:21:14.266731 14138 raft_consensus.cc:482] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:21:14.266834 14138 raft_consensus.cc:3037] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:14.269179 14138 raft_consensus.cc:504] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbb871491ad41d8b142074d92ae9e6b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 40233 } } peers { permanent_uuid: "43b0420fd0d8437d96c0594df6c1d57d" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 41839 } } peers { permanent_uuid: "ac5052f67e2d438e8399b8ee2e211075" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 45193 } }
I20260929 20:21:14.269793 14138 leader_election.cc:288] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b [CANDIDATE]: Term 1 election: Requested vote from peers 43b0420fd0d8437d96c0594df6c1d57d (127.27.185.67:41839), ac5052f67e2d438e8399b8ee2e211075 (127.27.185.65:45193)
I20260929 20:21:14.270270 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628d1a5fdb5a4f96b96c80fe11832d09" candidate_uuid: "9fbb871491ad41d8b142074d92ae9e6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac5052f67e2d438e8399b8ee2e211075"
I20260929 20:21:14.270270 14068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628d1a5fdb5a4f96b96c80fe11832d09" candidate_uuid: "9fbb871491ad41d8b142074d92ae9e6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43b0420fd0d8437d96c0594df6c1d57d"
I20260929 20:21:14.270489 14068 raft_consensus.cc:3037] T 628d1a5fdb5a4f96b96c80fe11832d09 P 43b0420fd0d8437d96c0594df6c1d57d [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:14.270500 13807 raft_consensus.cc:3037] T 628d1a5fdb5a4f96b96c80fe11832d09 P ac5052f67e2d438e8399b8ee2e211075 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:14.273789 14068 raft_consensus.cc:2454] T 628d1a5fdb5a4f96b96c80fe11832d09 P 43b0420fd0d8437d96c0594df6c1d57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fbb871491ad41d8b142074d92ae9e6b in term 1.
I20260929 20:21:14.273789 13807 raft_consensus.cc:2454] T 628d1a5fdb5a4f96b96c80fe11832d09 P ac5052f67e2d438e8399b8ee2e211075 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fbb871491ad41d8b142074d92ae9e6b in term 1.
I20260929 20:21:14.274255 13871 leader_election.cc:302] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43b0420fd0d8437d96c0594df6c1d57d, 9fbb871491ad41d8b142074d92ae9e6b; no voters:
I20260929 20:21:14.274571 14138 raft_consensus.cc:2781] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:21:14.275085 14138 raft_consensus.cc:686] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b [term 1 LEADER]: Becoming Leader. State: Replica: 9fbb871491ad41d8b142074d92ae9e6b, State: Running, Role: LEADER
I20260929 20:21:14.275730 14138 consensus_queue.cc:237] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbb871491ad41d8b142074d92ae9e6b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 40233 } } peers { permanent_uuid: "43b0420fd0d8437d96c0594df6c1d57d" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 41839 } } peers { permanent_uuid: "ac5052f67e2d438e8399b8ee2e211075" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 45193 } }
I20260929 20:21:14.280613 13666 catalog_manager.cc:5495] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fbb871491ad41d8b142074d92ae9e6b (127.27.185.66). New cstate: current_term: 1 leader_uuid: "9fbb871491ad41d8b142074d92ae9e6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbb871491ad41d8b142074d92ae9e6b" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 40233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43b0420fd0d8437d96c0594df6c1d57d" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 41839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac5052f67e2d438e8399b8ee2e211075" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 45193 } health_report { overall_health: UNKNOWN } } }
W20260929 20:21:14.281862 14114 tablet.cc:2369] T 628d1a5fdb5a4f96b96c80fe11832d09 P 43b0420fd0d8437d96c0594df6c1d57d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:21:14.293375 13984 tablet.cc:2369] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:21:14.330583 13853 tablet.cc:2369] T 628d1a5fdb5a4f96b96c80fe11832d09 P ac5052f67e2d438e8399b8ee2e211075: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:21:14.333410 14126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:14.328398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57110 (local address 127.27.185.66:40233)
0929 20:21:14.328608 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:14.328612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:14.328630 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:21:14.328776 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:14.328780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:14.328879 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:14.329004 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:21:14.329298 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.329998 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.331421 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.331839 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.331938 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:14.332363 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:21:14.332863 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:21:14.332972 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:14.333042 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 20:21:14.333104 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 20:21:14.341807 13807 raft_consensus.cc:1261] T 628d1a5fdb5a4f96b96c80fe11832d09 P ac5052f67e2d438e8399b8ee2e211075 [term 1 FOLLOWER]: Refusing update from remote peer 9fbb871491ad41d8b142074d92ae9e6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:21:14.341809 14068 raft_consensus.cc:1261] T 628d1a5fdb5a4f96b96c80fe11832d09 P 43b0420fd0d8437d96c0594df6c1d57d [term 1 FOLLOWER]: Refusing update from remote peer 9fbb871491ad41d8b142074d92ae9e6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:21:14.342922 14138 consensus_queue.cc:1035] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac5052f67e2d438e8399b8ee2e211075" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:21:14.343258 14142 consensus_queue.cc:1035] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "43b0420fd0d8437d96c0594df6c1d57d" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 41839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:21:14.353183 14145 mvcc.cc:205] Tried to move back new op lower bound from 7334761571693453312 to 7334761571437551616. Current Snapshot: MvccSnapshot[applied={T|T < 7334761571693453312}]
I20260929 20:21:14.354960 14146 mvcc.cc:205] Tried to move back new op lower bound from 7334761571693453312 to 7334761571437551616. Current Snapshot: MvccSnapshot[applied={T|T < 7334761571693453312}]
I20260929 20:21:14.373427 13665 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:21:14.373718 13665 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:21:14.377660 13665 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 654d93e0f288457d943e836fb1a25476: Sending DeleteTablet for 3 replicas of tablet 628d1a5fdb5a4f96b96c80fe11832d09
I20260929 20:21:14.378993 14048 tablet_service.cc:1514] Processing DeleteTablet for tablet 628d1a5fdb5a4f96b96c80fe11832d09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:21:14 UTC) from {username='slave'} at 127.0.0.1:52074
I20260929 20:21:14.379048 13918 tablet_service.cc:1514] Processing DeleteTablet for tablet 628d1a5fdb5a4f96b96c80fe11832d09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:21:14 UTC) from {username='slave'} at 127.0.0.1:57100
I20260929 20:21:14.380288 13787 tablet_service.cc:1514] Processing DeleteTablet for tablet 628d1a5fdb5a4f96b96c80fe11832d09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:21:14 UTC) from {username='slave'} at 127.0.0.1:52502
I20260929 20:21:14.382334 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 13725
I20260929 20:21:14.395993 14157 tablet_replica.cc:330] T 628d1a5fdb5a4f96b96c80fe11832d09 P ac5052f67e2d438e8399b8ee2e211075: stopping tablet replica
I20260929 20:21:14.396548 14157 raft_consensus.cc:2229] T 628d1a5fdb5a4f96b96c80fe11832d09 P ac5052f67e2d438e8399b8ee2e211075 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:21:14.397127 14157 raft_consensus.cc:2258] T 628d1a5fdb5a4f96b96c80fe11832d09 P ac5052f67e2d438e8399b8ee2e211075 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:21:14.400197 14157 ts_tablet_manager.cc:1905] T 628d1a5fdb5a4f96b96c80fe11832d09 P ac5052f67e2d438e8399b8ee2e211075: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:21:14.401386 14155 tablet_replica.cc:330] T 628d1a5fdb5a4f96b96c80fe11832d09 P 43b0420fd0d8437d96c0594df6c1d57d: stopping tablet replica
I20260929 20:21:14.401865 14155 raft_consensus.cc:2229] T 628d1a5fdb5a4f96b96c80fe11832d09 P 43b0420fd0d8437d96c0594df6c1d57d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:21:14.402459 14155 raft_consensus.cc:2258] T 628d1a5fdb5a4f96b96c80fe11832d09 P 43b0420fd0d8437d96c0594df6c1d57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:21:14.405263 14130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:14.383926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52504 (local address 127.27.185.65:45193)
0929 20:21:14.384115 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:14.384122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:14.388610 (+ 4488us) server_negotiation.cc:407] Connection header received
0929 20:21:14.389549 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:14.389556 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:14.389692 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:14.389780 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:14.391000 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.392265 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.398366 (+ 6101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.399074 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.399193 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:14.403370 (+ 4177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:14.403402 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:14.403421 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:14.404236 (+ 815us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:14.404301 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:14.404336 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:14.404432 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:14.404714 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 20:21:14.404842 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 20:21:14.406544 13847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:14.415199 14156 tablet_replica.cc:330] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b: stopping tablet replica
I20260929 20:21:14.415733 14156 raft_consensus.cc:2229] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:21:14.416497 14156 raft_consensus.cc:2258] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:21:14.416512 14155 ts_tablet_manager.cc:1905] T 628d1a5fdb5a4f96b96c80fe11832d09 P 43b0420fd0d8437d96c0594df6c1d57d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:21:14.418691 14156 ts_tablet_manager.cc:1905] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:21:14.426641 14155 ts_tablet_manager.cc:1918] T 628d1a5fdb5a4f96b96c80fe11832d09 P 43b0420fd0d8437d96c0594df6c1d57d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:21:14.426976 14155 log.cc:1196] T 628d1a5fdb5a4f96b96c80fe11832d09 P 43b0420fd0d8437d96c0594df6c1d57d: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-2/wal/wals/628d1a5fdb5a4f96b96c80fe11832d09
I20260929 20:21:14.427657 14155 ts_tablet_manager.cc:1939] T 628d1a5fdb5a4f96b96c80fe11832d09 P 43b0420fd0d8437d96c0594df6c1d57d: Deleting consensus metadata
I20260929 20:21:14.428117 14156 ts_tablet_manager.cc:1918] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:21:14.428298 14156 log.cc:1196] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790712966917858-28389-0/minicluster-data/ts-1/wal/wals/628d1a5fdb5a4f96b96c80fe11832d09
I20260929 20:21:14.428869 14156 ts_tablet_manager.cc:1939] T 628d1a5fdb5a4f96b96c80fe11832d09 P 9fbb871491ad41d8b142074d92ae9e6b: Deleting consensus metadata
I20260929 20:21:14.429867 13650 catalog_manager.cc:4841] TS 43b0420fd0d8437d96c0594df6c1d57d (127.27.185.67:41839): tablet 628d1a5fdb5a4f96b96c80fe11832d09 (table test-table [id=c73d46d69fe34f8cbc1e233cb1547f3c]) successfully deleted
I20260929 20:21:14.430709 13651 catalog_manager.cc:4841] TS 9fbb871491ad41d8b142074d92ae9e6b (127.27.185.66:40233): tablet 628d1a5fdb5a4f96b96c80fe11832d09 (table test-table [id=c73d46d69fe34f8cbc1e233cb1547f3c]) successfully deleted
I20260929 20:21:14.520648 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 13725
I20260929 20:21:14.539984 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 13856
W20260929 20:21:14.540642 13652 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.185.65:45193 (error 108)
I20260929 20:21:14.542713 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:14.542186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:45193 (local address 127.0.0.1:52512)
0929 20:21:14.542411 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:14.542545 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260929 20:21:14.543023 13652 catalog_manager.cc:4584] TS ac5052f67e2d438e8399b8ee2e211075 (127.27.185.65:45193): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 628d1a5fdb5a4f96b96c80fe11832d09: Network error: Client connection negotiation failed: client connection to 127.27.185.65:45193: connect: Connection refused (error 111)
I20260929 20:21:14.548238 14126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:14.541135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57124 (local address 127.27.185.66:40233)
0929 20:21:14.541342 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:14.541348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:14.541371 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:21:14.541582 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:14.541588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:14.541689 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:14.541811 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:14.542438 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.543500 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.544949 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.545389 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.545524 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:14.546578 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:14.546593 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:14.546606 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:14.547206 (+ 600us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:14.547256 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:14.547275 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:14.547461 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:14.547727 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 20:21:14.547824 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 20:21:14.548918 13978 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:14.596715 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:14.596127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:45193 (local address 127.0.0.1:52514)
0929 20:21:14.596383 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:14.596511 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 20:21:14.597780 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:14.597401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:45193 (local address 127.0.0.1:52530)
0929 20:21:14.597548 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:14.597632 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 20:21:14.658465 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:14.657884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:45193 (local address 127.0.0.1:52538)
0929 20:21:14.658135 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:14.658261 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 20:21:14.659680 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:14.659256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:45193 (local address 127.0.0.1:52548)
0929 20:21:14.659443 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:14.659533 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 20:21:14.664865 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 13856
I20260929 20:21:14.683569 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 13986
I20260929 20:21:14.690481 14129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:14.684311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52082 (local address 127.27.185.67:41839)
0929 20:21:14.684509 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:14.684514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:14.684531 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:21:14.684625 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:14.684629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:14.684712 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:14.684794 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:14.685196 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.685905 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.687210 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:14.687787 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:14.687906 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:14.689016 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:14.689025 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:14.689041 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:14.689666 (+ 625us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:14.689720 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:14.689741 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:14.689830 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:14.689989 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 20:21:14.690089 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:21:14.691239 14108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:14.740373 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:14.739794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:45193 (local address 127.0.0.1:52552)
0929 20:21:14.740044 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:14.740165 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 20:21:14.741494 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:14.741077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:45193 (local address 127.0.0.1:52562)
0929 20:21:14.741278 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:14.741362 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260929 20:21:14.781355 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 13986
I20260929 20:21:14.798343 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 13635
I20260929 20:21:14.799575 13696 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:14.909089 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 13635
2026-09-29T20:21:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2957 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:21:15 dist-test-slave-4cx3 krb5kdc[14166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:21:15 dist-test-slave-4cx3 krb5kdc[14166](info): set up 2 sockets
Sep 29 20:21:15 dist-test-slave-4cx3 krb5kdc[14166](info): commencing operation
krb5kdc: starting...
W20260929 20:21:16.059145 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:21:16 dist-test-slave-4cx3 krb5kdc[14166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:21:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:21:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
I20260929 20:21:16.306247 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:42629
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:32919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:42629
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:21:16.646238 14179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:16.646565 14179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:16.646729 14179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:16.655592 14179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:21:16.655716 14179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:16.655792 14179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:16.655891 14179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:21:16.655956 14179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:21:16.666729 14179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:32919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:42629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:42629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:16.668491 14179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:16.669950 14179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:16.682340 14184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:16.682376 14185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:16.682427 14187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:16.682965 14179 server_base.cc:1034] running on GCE node
I20260929 20:21:16.683846 14179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:16.685335 14179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:16.686539 14179 hybrid_clock.cc:648] HybridClock initialized: now 1790713276686469 us; error 59 us; skew 500 ppm
I20260929 20:21:16.686889 14179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:21:16 dist-test-slave-4cx3 krb5kdc[14166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713276, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:21:16.690107 14179 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:21:16.691782 14179 webserver.cc:460] Webserver started at http://127.27.185.126:42145/ using document root <none> and password file <none>
I20260929 20:21:16.692497 14179 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:16.692646 14179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:16.692970 14179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:16.696163 14179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "54dfd668b57f4814af9ae701b33e1f44"
format_stamp: "Formatted at 2026-09-29 20:21:16 on dist-test-slave-4cx3"
server_key: "6a02431741cd5e5a6d77f0c5c6660e38"
server_key_iv: "526aab3bfdf99b9afa4df8e303a82381"
server_key_version: "encryptionkey@0"
I20260929 20:21:16.696943 14179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "54dfd668b57f4814af9ae701b33e1f44"
format_stamp: "Formatted at 2026-09-29 20:21:16 on dist-test-slave-4cx3"
server_key: "6a02431741cd5e5a6d77f0c5c6660e38"
server_key_iv: "526aab3bfdf99b9afa4df8e303a82381"
server_key_version: "encryptionkey@0"
I20260929 20:21:16.702255 14179 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:21:16.705772 14194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:16.707105 14179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 20:21:16.707294 14179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "54dfd668b57f4814af9ae701b33e1f44"
format_stamp: "Formatted at 2026-09-29 20:21:16 on dist-test-slave-4cx3"
server_key: "6a02431741cd5e5a6d77f0c5c6660e38"
server_key_iv: "526aab3bfdf99b9afa4df8e303a82381"
server_key_version: "encryptionkey@0"
I20260929 20:21:16.707543 14179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:16.730736 14179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:16.735421 14179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:16.735726 14179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:16.754796 14179 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:42629
I20260929 20:21:16.754827 14245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:42629 every 8 connection(s)
I20260929 20:21:16.756385 14179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:21:16.760365 14246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:16.760903 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 14179
I20260929 20:21:16.761108 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:21:16.761720 28389 external_mini_cluster.cc:1428] Setting key 4028693d6be77470475ddaefec4c2412
I20260929 20:21:16.774751 14246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54dfd668b57f4814af9ae701b33e1f44: Bootstrap starting.
Sep 29 20:21:16 dist-test-slave-4cx3 krb5kdc[14166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:21:16.778491 14246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54dfd668b57f4814af9ae701b33e1f44: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:16.779630 14246 log.cc:824] T 00000000000000000000000000000000 P 54dfd668b57f4814af9ae701b33e1f44: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:16.782621 14246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54dfd668b57f4814af9ae701b33e1f44: No bootstrap required, opened a new log
I20260929 20:21:16.785600 14249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:16.763716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44818 (local address 127.27.185.126:42629)
0929 20:21:16.764524 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:16.764542 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:16.764583 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:21:16.766677 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:16.766700 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:16.767106 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:16.767500 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:16.769270 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:16.770575 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:16.772485 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:16.773157 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:16.773303 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:16.778130 (+ 4827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:16.778139 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:16.778154 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:16.778196 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:16.781374 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:16.782125 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:16.782132 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:16.782138 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:16.782311 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:16.782712 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:16.782731 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:16.782736 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:16.783282 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:16.783636 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:16.784073 (+ 437us) server_negotiation.cc:299] Negotiation successful
0929 20:21:16.784553 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":382,"threads_started":1}
I20260929 20:21:16.791456 14246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54dfd668b57f4814af9ae701b33e1f44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54dfd668b57f4814af9ae701b33e1f44" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 42629 } }
I20260929 20:21:16.791855 14246 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54dfd668b57f4814af9ae701b33e1f44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:16.791961 14246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54dfd668b57f4814af9ae701b33e1f44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54dfd668b57f4814af9ae701b33e1f44, State: Initialized, Role: FOLLOWER
I20260929 20:21:16.792514 14246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54dfd668b57f4814af9ae701b33e1f44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54dfd668b57f4814af9ae701b33e1f44" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 42629 } }
I20260929 20:21:16.792760 14246 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54dfd668b57f4814af9ae701b33e1f44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:21:16.792902 14246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54dfd668b57f4814af9ae701b33e1f44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:21:16.793072 14246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54dfd668b57f4814af9ae701b33e1f44 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:16.795259 14246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54dfd668b57f4814af9ae701b33e1f44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54dfd668b57f4814af9ae701b33e1f44" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 42629 } }
I20260929 20:21:16.795832 14246 leader_election.cc:302] T 00000000000000000000000000000000 P 54dfd668b57f4814af9ae701b33e1f44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54dfd668b57f4814af9ae701b33e1f44; no voters:
I20260929 20:21:16.796437 14246 leader_election.cc:288] T 00000000000000000000000000000000 P 54dfd668b57f4814af9ae701b33e1f44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:21:16.796595 14251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54dfd668b57f4814af9ae701b33e1f44 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:21:16.797597 14251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54dfd668b57f4814af9ae701b33e1f44 [term 1 LEADER]: Becoming Leader. State: Replica: 54dfd668b57f4814af9ae701b33e1f44, State: Running, Role: LEADER
I20260929 20:21:16.798223 14251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54dfd668b57f4814af9ae701b33e1f44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54dfd668b57f4814af9ae701b33e1f44" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 42629 } }
I20260929 20:21:16.798767 14246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54dfd668b57f4814af9ae701b33e1f44 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:21:16.801182 14252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54dfd668b57f4814af9ae701b33e1f44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54dfd668b57f4814af9ae701b33e1f44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54dfd668b57f4814af9ae701b33e1f44" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 42629 } } }
I20260929 20:21:16.801591 14252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54dfd668b57f4814af9ae701b33e1f44 [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:16.801134 14253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54dfd668b57f4814af9ae701b33e1f44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54dfd668b57f4814af9ae701b33e1f44. Latest consensus state: current_term: 1 leader_uuid: "54dfd668b57f4814af9ae701b33e1f44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54dfd668b57f4814af9ae701b33e1f44" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 42629 } } }
I20260929 20:21:16.802951 14253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54dfd668b57f4814af9ae701b33e1f44 [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:16.805235 14258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:21:16.810935 14258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:21:16.821103 14258 catalog_manager.cc:1347] Generated new cluster ID: d4a1fbb7d01d4c5bab3547cc19b2cb49
I20260929 20:21:16.821224 14258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:21:16.845187 14258 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:21:16.846602 14258 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:21:16.858742 14258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54dfd668b57f4814af9ae701b33e1f44: Generated new TSK 0
I20260929 20:21:16.859699 14258 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
I20260929 20:21:16.923658 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:0
--local_ip_for_outbound_sockets=127.27.185.65
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:42629
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:32919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:21:17.266729 14272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:17.267192 14272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:17.267412 14272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:17.275460 14272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:17.275650 14272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:17.275873 14272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:21:17.287011 14272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:32919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:42629
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:17.288408 14272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:17.289801 14272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:17.303455 14278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:17.303463 14280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:17.303700 14277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:17.304572 14272 server_base.cc:1034] running on GCE node
I20260929 20:21:17.305217 14272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:17.306146 14272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:17.307343 14272 hybrid_clock.cc:648] HybridClock initialized: now 1790713277307270 us; error 63 us; skew 500 ppm
I20260929 20:21:17.307754 14272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:21:17 dist-test-slave-4cx3 krb5kdc[14166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713277, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:21:17.310951 14272 init.cc:377] Logged in from keytab as kudu/127.27.185.65@KRBTEST.COM (short username kudu)
I20260929 20:21:17.312493 14272 webserver.cc:460] Webserver started at http://127.27.185.65:44205/ using document root <none> and password file <none>
I20260929 20:21:17.313210 14272 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:17.313340 14272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:17.313665 14272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:17.316692 14272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-0/data/instance:
uuid: "92cd131c34f0401a9cad6e0e2f0a2caf"
format_stamp: "Formatted at 2026-09-29 20:21:17 on dist-test-slave-4cx3"
server_key: "2081d2dc11ea704c3661c91b1f982bbe"
server_key_iv: "7eab5eebe5e231e9ef01206fc4abd4ec"
server_key_version: "encryptionkey@0"
I20260929 20:21:17.317468 14272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance:
uuid: "92cd131c34f0401a9cad6e0e2f0a2caf"
format_stamp: "Formatted at 2026-09-29 20:21:17 on dist-test-slave-4cx3"
server_key: "2081d2dc11ea704c3661c91b1f982bbe"
server_key_iv: "7eab5eebe5e231e9ef01206fc4abd4ec"
server_key_version: "encryptionkey@0"
I20260929 20:21:17.322978 14272 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:21:17.326768 14287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:17.328250 14272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:21:17.328424 14272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "92cd131c34f0401a9cad6e0e2f0a2caf"
format_stamp: "Formatted at 2026-09-29 20:21:17 on dist-test-slave-4cx3"
server_key: "2081d2dc11ea704c3661c91b1f982bbe"
server_key_iv: "7eab5eebe5e231e9ef01206fc4abd4ec"
server_key_version: "encryptionkey@0"
I20260929 20:21:17.328639 14272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:17.349471 14272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:17.353624 14272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:17.353933 14272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:17.354995 14272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:21:17.356544 14272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:21:17.356647 14272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:17.356803 14272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:21:17.356873 14272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:17.390928 14272 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:39173
I20260929 20:21:17.391014 14400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:39173 every 8 connection(s)
I20260929 20:21:17.392285 14272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:21:17.399572 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 14272
I20260929 20:21:17.399788 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance
I20260929 20:21:17.400388 28389 external_mini_cluster.cc:1428] Setting key 0aabf8f63bc05a661c4be33135b20194
Sep 29 20:21:17 dist-test-slave-4cx3 krb5kdc[14166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713277, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:21:17.414945 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:17.394953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:35767 (local address 127.27.185.126:42629)
0929 20:21:17.395463 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:17.395468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:17.396399 (+ 931us) server_negotiation.cc:407] Connection header received
0929 20:21:17.398236 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:17.398242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:17.398359 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:17.398481 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:17.400263 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:17.401176 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:17.403195 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:17.403673 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:17.403812 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:17.408057 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:17.408062 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:17.408065 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:17.408098 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:17.411195 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:17.412190 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:17.412199 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:17.412206 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:17.412359 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:17.412829 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:17.412837 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:17.412844 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:17.413252 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:17.413419 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:17.414174 (+ 755us) server_negotiation.cc:299] Negotiation successful
0929 20:21:17.414444 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":252,"threads_started":1}
I20260929 20:21:17.415634 14404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:17.395499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:42629 (local address 127.27.185.65:35767)
0929 20:21:17.396231 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:17.396276 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:17.397962 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:17.398751 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:17.398760 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:17.399337 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:21:17.400093 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:17.400114 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:17.401344 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:17.401350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:17.403053 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:17.403065 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:17.403911 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:17.403919 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:17.404179 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:17.404874 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:21:17.404902 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:21:17.407797 (+ 2895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:21:17.411495 (+ 3698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:21:17.411503 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:21:17.411518 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:21:17.412049 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:21:17.412503 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:21:17.412510 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:21:17.412513 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:21:17.412712 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:21:17.413379 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:21:17.413385 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:21:17.413756 (+ 371us) client_negotiation.cc:769] Sending connection context
0929 20:21:17.414000 (+ 244us) client_negotiation.cc:240] Negotiation successful
0929 20:21:17.414416 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":276,"threads_started":1}
I20260929 20:21:17.417320 14401 heartbeater.cc:347] Connected to a master server at 127.27.185.126:42629
I20260929 20:21:17.417641 14401 heartbeater.cc:464] Registering TS with master...
I20260929 20:21:17.418380 14401 heartbeater.cc:511] Master 127.27.185.126:42629 requested a full tablet report, sending...
I20260929 20:21:17.420558 14211 ts_manager.cc:194] Registered new tserver with Master: 92cd131c34f0401a9cad6e0e2f0a2caf (127.27.185.65:39173)
I20260929 20:21:17.422106 14211 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.65@KRBTEST.COM'} at 127.27.185.65:35767
WARNING: no policy specified for kudu/127.27.185.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
I20260929 20:21:17.459311 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:0
--local_ip_for_outbound_sockets=127.27.185.66
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:42629
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:32919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:21:17.801110 14407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:17.801425 14407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:17.801613 14407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:17.809161 14407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:17.809320 14407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:17.809512 14407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:21:17.819655 14407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:32919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:42629
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:17.821019 14407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:17.822402 14407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:17.836606 14413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:17.836606 14412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:17.837347 14407 server_base.cc:1034] running on GCE node
W20260929 20:21:17.837379 14415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:17.838063 14407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:17.839028 14407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:17.840253 14407 hybrid_clock.cc:648] HybridClock initialized: now 1790713277840195 us; error 48 us; skew 500 ppm
I20260929 20:21:17.840593 14407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:21:17 dist-test-slave-4cx3 krb5kdc[14166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713277, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:21:17.843878 14407 init.cc:377] Logged in from keytab as kudu/127.27.185.66@KRBTEST.COM (short username kudu)
I20260929 20:21:17.845404 14407 webserver.cc:460] Webserver started at http://127.27.185.66:45245/ using document root <none> and password file <none>
I20260929 20:21:17.846127 14407 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:17.846268 14407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:17.846649 14407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:17.849782 14407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-1/data/instance:
uuid: "17f2fc9fd28647e7b2969e41cf2767de"
format_stamp: "Formatted at 2026-09-29 20:21:17 on dist-test-slave-4cx3"
server_key: "9316f9edc247ddf7d4012dc7b8ae2727"
server_key_iv: "104e8b755892ff1298608d6d151151a6"
server_key_version: "encryptionkey@0"
I20260929 20:21:17.850566 14407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance:
uuid: "17f2fc9fd28647e7b2969e41cf2767de"
format_stamp: "Formatted at 2026-09-29 20:21:17 on dist-test-slave-4cx3"
server_key: "9316f9edc247ddf7d4012dc7b8ae2727"
server_key_iv: "104e8b755892ff1298608d6d151151a6"
server_key_version: "encryptionkey@0"
I20260929 20:21:17.856009 14407 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:21:17.859964 14422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:17.861513 14407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:21:17.861721 14407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "17f2fc9fd28647e7b2969e41cf2767de"
format_stamp: "Formatted at 2026-09-29 20:21:17 on dist-test-slave-4cx3"
server_key: "9316f9edc247ddf7d4012dc7b8ae2727"
server_key_iv: "104e8b755892ff1298608d6d151151a6"
server_key_version: "encryptionkey@0"
I20260929 20:21:17.861940 14407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:17.877656 14407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:17.882616 14407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:17.882934 14407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:17.884054 14407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:21:17.885519 14407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:21:17.885627 14407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:17.885767 14407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:21:17.885849 14407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:17.919689 14407 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:34783
I20260929 20:21:17.919735 14534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:34783 every 8 connection(s)
I20260929 20:21:17.921065 14407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:21:17.924721 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 14407
I20260929 20:21:17.924924 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance
I20260929 20:21:17.925458 28389 external_mini_cluster.cc:1428] Setting key b93cd3c7e86df7ddfe2b07ed92840d0d
Sep 29 20:21:17 dist-test-slave-4cx3 krb5kdc[14166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713277, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:21:17.942284 14537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:17.923632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:53875 (local address 127.27.185.126:42629)
0929 20:21:17.924075 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:17.924081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:17.925300 (+ 1219us) server_negotiation.cc:407] Connection header received
0929 20:21:17.927374 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:17.927380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:17.927480 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:17.927588 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:17.929151 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:17.930353 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:17.931989 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:17.932820 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:17.932970 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:17.936645 (+ 3675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:17.936650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:17.936653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:17.936684 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:17.939221 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:17.939991 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:17.939995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:17.939998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:17.940122 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:17.940550 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:17.940554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:17.940556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:17.940818 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:17.940998 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:17.941673 (+ 675us) server_negotiation.cc:299] Negotiation successful
0929 20:21:17.941899 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":220,"threads_started":1}
I20260929 20:21:17.943329 14538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:17.924260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:42629 (local address 127.27.185.66:53875)
0929 20:21:17.925132 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:17.925177 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:17.927035 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:17.927811 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:17.927820 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:17.928386 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:21:17.928965 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:17.928982 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:17.930486 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:17.930489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:17.931805 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:17.931817 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:17.933062 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:17.933068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:17.933276 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:17.934052 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:21:17.934076 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:21:17.936411 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:21:17.939378 (+ 2967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:21:17.939414 (+ 36us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:21:17.939432 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:21:17.939858 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:21:17.940229 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:21:17.940237 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:21:17.940240 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:21:17.940425 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:21:17.940947 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:21:17.940955 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:21:17.941319 (+ 364us) client_negotiation.cc:769] Sending connection context
0929 20:21:17.941582 (+ 263us) client_negotiation.cc:240] Negotiation successful
0929 20:21:17.942138 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":317,"threads_started":1}
I20260929 20:21:17.944835 14535 heartbeater.cc:347] Connected to a master server at 127.27.185.126:42629
I20260929 20:21:17.945195 14535 heartbeater.cc:464] Registering TS with master...
I20260929 20:21:17.946098 14535 heartbeater.cc:511] Master 127.27.185.126:42629 requested a full tablet report, sending...
I20260929 20:21:17.947875 14211 ts_manager.cc:194] Registered new tserver with Master: 17f2fc9fd28647e7b2969e41cf2767de (127.27.185.66:34783)
I20260929 20:21:17.948729 14211 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.66@KRBTEST.COM'} at 127.27.185.66:53875
WARNING: no policy specified for kudu/127.27.185.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
I20260929 20:21:17.984988 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:0
--local_ip_for_outbound_sockets=127.27.185.67
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:42629
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:32919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:21:18.315325 14541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:18.315655 14541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:18.315853 14541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:18.323189 14541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:18.323314 14541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:18.323536 14541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:21:18.333837 14541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:32919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:42629
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:18.335160 14541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:18.336547 14541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:18.350570 14547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:18.350580 14546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:18.351079 14541 server_base.cc:1034] running on GCE node
W20260929 20:21:18.351137 14549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:18.351876 14541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:18.352869 14541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:18.354090 14541 hybrid_clock.cc:648] HybridClock initialized: now 1790713278354045 us; error 57 us; skew 500 ppm
I20260929 20:21:18.354410 14541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:21:18 dist-test-slave-4cx3 krb5kdc[14166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713278, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:21:18.357728 14541 init.cc:377] Logged in from keytab as kudu/127.27.185.67@KRBTEST.COM (short username kudu)
I20260929 20:21:18.359443 14541 webserver.cc:460] Webserver started at http://127.27.185.67:37185/ using document root <none> and password file <none>
I20260929 20:21:18.360142 14541 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:18.360246 14541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:18.360507 14541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:18.363502 14541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-2/data/instance:
uuid: "c0f9a4f342414eb08754da901064a2b1"
format_stamp: "Formatted at 2026-09-29 20:21:18 on dist-test-slave-4cx3"
server_key: "974e8a07da2144ad71bbf49518a35315"
server_key_iv: "258e4b649b1109c3c6560897424ec318"
server_key_version: "encryptionkey@0"
I20260929 20:21:18.364210 14541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance:
uuid: "c0f9a4f342414eb08754da901064a2b1"
format_stamp: "Formatted at 2026-09-29 20:21:18 on dist-test-slave-4cx3"
server_key: "974e8a07da2144ad71bbf49518a35315"
server_key_iv: "258e4b649b1109c3c6560897424ec318"
server_key_version: "encryptionkey@0"
I20260929 20:21:18.369792 14541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260929 20:21:18.373412 14556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:18.374774 14541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 20:21:18.374953 14541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "c0f9a4f342414eb08754da901064a2b1"
format_stamp: "Formatted at 2026-09-29 20:21:18 on dist-test-slave-4cx3"
server_key: "974e8a07da2144ad71bbf49518a35315"
server_key_iv: "258e4b649b1109c3c6560897424ec318"
server_key_version: "encryptionkey@0"
I20260929 20:21:18.375162 14541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:18.390841 14541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:18.395323 14541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:18.395664 14541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:18.396651 14541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:21:18.397957 14541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:21:18.398036 14541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:18.398193 14541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:21:18.398242 14541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:18.425112 14401 heartbeater.cc:503] Master 127.27.185.126:42629 was elected leader, sending a full tablet report...
I20260929 20:21:18.430332 14541 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:40833
I20260929 20:21:18.430364 14668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:40833 every 8 connection(s)
I20260929 20:21:18.431891 14541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:21:18.440101 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 14541
I20260929 20:21:18.440294 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance
I20260929 20:21:18.440810 28389 external_mini_cluster.cc:1428] Setting key bd64a02df00b6e875b91debf3289793f
Sep 29 20:21:18 dist-test-slave-4cx3 krb5kdc[14166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713278, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:21:18.452198 14537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:18.434510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:54717 (local address 127.27.185.126:42629)
0929 20:21:18.434685 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:18.434690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:18.435986 (+ 1296us) server_negotiation.cc:407] Connection header received
0929 20:21:18.437805 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:18.437809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:18.437879 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:18.437967 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:18.439610 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.440728 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.442770 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.443169 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.443264 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:18.446771 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:18.446776 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:18.446780 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:18.446825 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:18.449456 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:18.450197 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:18.450200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:18.450203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:18.450319 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:18.450693 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:18.450696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:18.450698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:18.450956 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:18.451081 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:18.451783 (+ 702us) server_negotiation.cc:299] Negotiation successful
0929 20:21:18.451961 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:21:18.453244 14671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:18.435060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:42629 (local address 127.27.185.67:54717)
0929 20:21:18.435817 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:18.435859 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:18.437506 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:18.438187 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:18.438195 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:18.438775 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:21:18.439385 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:18.439457 (+ 72us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.440898 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.440906 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:18.442643 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:18.442654 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.443419 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.443428 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:18.443753 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:18.444404 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:21:18.444435 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:21:18.446576 (+ 2141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:21:18.449584 (+ 3008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:21:18.449591 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:21:18.449605 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:21:18.450093 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:21:18.450403 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:21:18.450406 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:21:18.450408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:21:18.450600 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:21:18.451057 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:21:18.451062 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:21:18.451424 (+ 362us) client_negotiation.cc:769] Sending connection context
0929 20:21:18.451695 (+ 271us) client_negotiation.cc:240] Negotiation successful
0929 20:21:18.452093 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":323,"threads_started":1}
I20260929 20:21:18.454917 14669 heartbeater.cc:347] Connected to a master server at 127.27.185.126:42629
I20260929 20:21:18.455262 14669 heartbeater.cc:464] Registering TS with master...
I20260929 20:21:18.456142 14669 heartbeater.cc:511] Master 127.27.185.126:42629 requested a full tablet report, sending...
I20260929 20:21:18.457486 14211 ts_manager.cc:194] Registered new tserver with Master: c0f9a4f342414eb08754da901064a2b1 (127.27.185.67:40833)
I20260929 20:21:18.458256 14211 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.67@KRBTEST.COM'} at 127.27.185.67:54717
I20260929 20:21:18.468451 28389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:21:18.485456 14537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:18.472648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44822 (local address 127.27.185.126:42629)
0929 20:21:18.472816 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:18.472821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:18.473309 (+ 488us) server_negotiation.cc:407] Connection header received
0929 20:21:18.473457 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:18.473461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:18.473542 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:18.473629 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:18.475059 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.475862 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.477247 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.477641 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.477732 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:18.480555 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:18.480561 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:18.480564 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:18.480598 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:18.482932 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:18.483747 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:18.483758 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:18.483761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:18.483889 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:18.484256 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:18.484259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:18.484261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:18.484546 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:18.484750 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:18.485071 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 20:21:18.485206 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:21:18.489403 14211 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:21:18.496374 14211 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:21:18.528286 14681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:18.508507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:40833 (local address 127.0.0.1:53944)
0929 20:21:18.515587 (+ 7080us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:18.515646 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:18.515905 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:18.517369 (+ 1464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:18.517380 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:18.517445 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:21:18.518026 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:18.518043 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.520799 (+ 2756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.520808 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:18.524177 (+ 3369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:18.524201 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.526585 (+ 2384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.526591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:18.526805 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:18.526870 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 20:21:18.527621 (+ 751us) client_negotiation.cc:240] Negotiation successful
0929 20:21:18.527866 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6687,"thread_start_us":7998,"threads_started":1}
I20260929 20:21:18.529379 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:18.508885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53944 (local address 127.27.185.67:40833)
0929 20:21:18.514236 (+ 5351us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:18.514249 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:18.516195 (+ 1946us) server_negotiation.cc:407] Connection header received
0929 20:21:18.516346 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:18.516359 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:18.516725 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:18.516980 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:21:18.518308 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.520560 (+ 2252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.524448 (+ 3888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.526376 (+ 1928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.526529 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:18.526807 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:18.528555 (+ 1748us) server_negotiation.cc:299] Negotiation successful
0929 20:21:18.529013 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5104,"thread_start_us":5686,"threads_started":1}
I20260929 20:21:18.533649 14604 tablet_service.cc:1467] Processing CreateTablet for tablet 065507805aae403da1ea23233ec27d67 (DEFAULT_TABLE table=test-table [id=0b040bf87f7b458cbc6f925b4c675bea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:21:18.534621 14683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:18.510648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:34783 (local address 127.0.0.1:42616)
0929 20:21:18.526432 (+ 15784us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:18.526453 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:18.526579 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:18.527454 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:18.527458 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:18.527487 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:21:18.527933 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:18.527944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.529830 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.529835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:18.531943 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:18.531959 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.533798 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.533804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:18.533985 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:18.534019 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 20:21:18.534116 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 20:21:18.534259 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15652,"thread_start_us":15077,"threads_started":1}
I20260929 20:21:18.534910 14686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:18.510787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42616 (local address 127.27.185.66:34783)
0929 20:21:18.525015 (+ 14228us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:18.525037 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:18.526715 (+ 1678us) server_negotiation.cc:407] Connection header received
0929 20:21:18.526828 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:18.526838 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:18.527119 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:18.527320 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:21:18.528138 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.529687 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.532145 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.533590 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.533740 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:18.533941 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:18.534196 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 20:21:18.534366 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14072,"thread_start_us":13954,"threads_started":1}
I20260929 20:21:18.535576 14604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 065507805aae403da1ea23233ec27d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:18.535844 14685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:18.509542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:39173 (local address 127.0.0.1:44684)
0929 20:21:18.526137 (+ 16595us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:18.526161 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:18.526316 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:18.527074 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:18.527080 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:18.527103 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:21:18.527623 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:18.527638 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.529574 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.529579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:18.532954 (+ 3375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:18.532966 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.534967 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.534972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:18.535170 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:18.535203 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:21:18.535300 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 20:21:18.535456 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16396,"mutex_wait_us":1194,"thread_start_us":16600,"threads_started":1}
I20260929 20:21:18.536172 14684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:18.510136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44684 (local address 127.27.185.65:39173)
0929 20:21:18.525567 (+ 15431us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:18.525579 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:18.526402 (+ 823us) server_negotiation.cc:407] Connection header received
0929 20:21:18.526535 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:18.526548 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:18.526834 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:18.527058 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:21:18.527821 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.529338 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.533121 (+ 3783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.534437 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.534587 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:18.534831 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:18.535372 (+ 541us) server_negotiation.cc:299] Negotiation successful
0929 20:21:18.535554 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15235,"thread_start_us":15075,"threads_started":1}
I20260929 20:21:18.538859 14470 tablet_service.cc:1467] Processing CreateTablet for tablet 065507805aae403da1ea23233ec27d67 (DEFAULT_TABLE table=test-table [id=0b040bf87f7b458cbc6f925b4c675bea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:21:18.539170 14335 tablet_service.cc:1467] Processing CreateTablet for tablet 065507805aae403da1ea23233ec27d67 (DEFAULT_TABLE table=test-table [id=0b040bf87f7b458cbc6f925b4c675bea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:21:18.540271 14470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 065507805aae403da1ea23233ec27d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:18.540463 14335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 065507805aae403da1ea23233ec27d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:18.551298 14687 tablet_bootstrap.cc:492] T 065507805aae403da1ea23233ec27d67 P c0f9a4f342414eb08754da901064a2b1: Bootstrap starting.
I20260929 20:21:18.553380 14688 tablet_bootstrap.cc:492] T 065507805aae403da1ea23233ec27d67 P 92cd131c34f0401a9cad6e0e2f0a2caf: Bootstrap starting.
I20260929 20:21:18.555047 14689 tablet_bootstrap.cc:492] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de: Bootstrap starting.
I20260929 20:21:18.556869 14688 tablet_bootstrap.cc:654] T 065507805aae403da1ea23233ec27d67 P 92cd131c34f0401a9cad6e0e2f0a2caf: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:18.557261 14687 tablet_bootstrap.cc:654] T 065507805aae403da1ea23233ec27d67 P c0f9a4f342414eb08754da901064a2b1: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:18.557958 14688 log.cc:824] T 065507805aae403da1ea23233ec27d67 P 92cd131c34f0401a9cad6e0e2f0a2caf: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:18.558496 14689 tablet_bootstrap.cc:654] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:18.558494 14687 log.cc:824] T 065507805aae403da1ea23233ec27d67 P c0f9a4f342414eb08754da901064a2b1: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:18.559921 14689 log.cc:824] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:18.561508 14688 tablet_bootstrap.cc:492] T 065507805aae403da1ea23233ec27d67 P 92cd131c34f0401a9cad6e0e2f0a2caf: No bootstrap required, opened a new log
I20260929 20:21:18.561800 14688 ts_tablet_manager.cc:1397] T 065507805aae403da1ea23233ec27d67 P 92cd131c34f0401a9cad6e0e2f0a2caf: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 20:21:18.561940 14687 tablet_bootstrap.cc:492] T 065507805aae403da1ea23233ec27d67 P c0f9a4f342414eb08754da901064a2b1: No bootstrap required, opened a new log
I20260929 20:21:18.562201 14687 ts_tablet_manager.cc:1397] T 065507805aae403da1ea23233ec27d67 P c0f9a4f342414eb08754da901064a2b1: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20260929 20:21:18.563113 14689 tablet_bootstrap.cc:492] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de: No bootstrap required, opened a new log
I20260929 20:21:18.563431 14689 ts_tablet_manager.cc:1397] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260929 20:21:18.570267 14687 raft_consensus.cc:348] T 065507805aae403da1ea23233ec27d67 P c0f9a4f342414eb08754da901064a2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f9a4f342414eb08754da901064a2b1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40833 } } peers { permanent_uuid: "92cd131c34f0401a9cad6e0e2f0a2caf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 39173 } } peers { permanent_uuid: "17f2fc9fd28647e7b2969e41cf2767de" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 34783 } }
I20260929 20:21:18.570778 14687 raft_consensus.cc:374] T 065507805aae403da1ea23233ec27d67 P c0f9a4f342414eb08754da901064a2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:18.570894 14687 raft_consensus.cc:729] T 065507805aae403da1ea23233ec27d67 P c0f9a4f342414eb08754da901064a2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f9a4f342414eb08754da901064a2b1, State: Initialized, Role: FOLLOWER
I20260929 20:21:18.571627 14687 consensus_queue.cc:260] T 065507805aae403da1ea23233ec27d67 P c0f9a4f342414eb08754da901064a2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f9a4f342414eb08754da901064a2b1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40833 } } peers { permanent_uuid: "92cd131c34f0401a9cad6e0e2f0a2caf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 39173 } } peers { permanent_uuid: "17f2fc9fd28647e7b2969e41cf2767de" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 34783 } }
I20260929 20:21:18.572826 14669 heartbeater.cc:503] Master 127.27.185.126:42629 was elected leader, sending a full tablet report...
I20260929 20:21:18.573477 14687 ts_tablet_manager.cc:1428] T 065507805aae403da1ea23233ec27d67 P c0f9a4f342414eb08754da901064a2b1: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260929 20:21:18.574421 14688 raft_consensus.cc:348] T 065507805aae403da1ea23233ec27d67 P 92cd131c34f0401a9cad6e0e2f0a2caf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f9a4f342414eb08754da901064a2b1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40833 } } peers { permanent_uuid: "92cd131c34f0401a9cad6e0e2f0a2caf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 39173 } } peers { permanent_uuid: "17f2fc9fd28647e7b2969e41cf2767de" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 34783 } }
I20260929 20:21:18.574559 14689 raft_consensus.cc:348] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f9a4f342414eb08754da901064a2b1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40833 } } peers { permanent_uuid: "92cd131c34f0401a9cad6e0e2f0a2caf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 39173 } } peers { permanent_uuid: "17f2fc9fd28647e7b2969e41cf2767de" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 34783 } }
I20260929 20:21:18.575239 14689 raft_consensus.cc:374] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:18.575280 14688 raft_consensus.cc:374] T 065507805aae403da1ea23233ec27d67 P 92cd131c34f0401a9cad6e0e2f0a2caf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:18.575459 14688 raft_consensus.cc:729] T 065507805aae403da1ea23233ec27d67 P 92cd131c34f0401a9cad6e0e2f0a2caf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cd131c34f0401a9cad6e0e2f0a2caf, State: Initialized, Role: FOLLOWER
I20260929 20:21:18.575357 14689 raft_consensus.cc:729] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17f2fc9fd28647e7b2969e41cf2767de, State: Initialized, Role: FOLLOWER
I20260929 20:21:18.576335 14688 consensus_queue.cc:260] T 065507805aae403da1ea23233ec27d67 P 92cd131c34f0401a9cad6e0e2f0a2caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f9a4f342414eb08754da901064a2b1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40833 } } peers { permanent_uuid: "92cd131c34f0401a9cad6e0e2f0a2caf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 39173 } } peers { permanent_uuid: "17f2fc9fd28647e7b2969e41cf2767de" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 34783 } }
I20260929 20:21:18.576382 14689 consensus_queue.cc:260] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f9a4f342414eb08754da901064a2b1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40833 } } peers { permanent_uuid: "92cd131c34f0401a9cad6e0e2f0a2caf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 39173 } } peers { permanent_uuid: "17f2fc9fd28647e7b2969e41cf2767de" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 34783 } }
I20260929 20:21:18.578485 14688 ts_tablet_manager.cc:1428] T 065507805aae403da1ea23233ec27d67 P 92cd131c34f0401a9cad6e0e2f0a2caf: Time spent starting tablet: real 0.017s user 0.012s sys 0.005s
I20260929 20:21:18.579788 14535 heartbeater.cc:503] Master 127.27.185.126:42629 was elected leader, sending a full tablet report...
I20260929 20:21:18.580369 14689 ts_tablet_manager.cc:1428] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de: Time spent starting tablet: real 0.017s user 0.013s sys 0.003s
I20260929 20:21:18.598554 14695 raft_consensus.cc:482] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:21:18.598891 14695 raft_consensus.cc:504] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f9a4f342414eb08754da901064a2b1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40833 } } peers { permanent_uuid: "92cd131c34f0401a9cad6e0e2f0a2caf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 39173 } } peers { permanent_uuid: "17f2fc9fd28647e7b2969e41cf2767de" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 34783 } }
I20260929 20:21:18.600811 14695 leader_election.cc:288] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f9a4f342414eb08754da901064a2b1 (127.27.185.67:40833), 92cd131c34f0401a9cad6e0e2f0a2caf (127.27.185.65:39173)
I20260929 20:21:18.610800 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:18.601255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:59539 (local address 127.27.185.67:40833)
0929 20:21:18.601503 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:18.601512 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:18.602535 (+ 1023us) server_negotiation.cc:407] Connection header received
0929 20:21:18.602768 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:18.602773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:18.602833 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:18.602943 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:21:18.603667 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.604873 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.608274 (+ 3401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.609555 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.609728 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:18.609806 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:18.610399 (+ 593us) server_negotiation.cc:299] Negotiation successful
0929 20:21:18.610551 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 20:21:18.610800 14684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:18.601312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:51739 (local address 127.27.185.65:39173)
0929 20:21:18.601649 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:18.601659 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:18.602556 (+ 897us) server_negotiation.cc:407] Connection header received
0929 20:21:18.602800 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:18.602807 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:18.602872 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:18.602999 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:21:18.603932 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.604995 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.608274 (+ 3279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.609555 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.609728 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:18.609806 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:18.610399 (+ 593us) server_negotiation.cc:299] Negotiation successful
0929 20:21:18.610536 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260929 20:21:18.611120 14697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:18.601141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:40833 (local address 127.27.185.66:59539)
0929 20:21:18.602399 (+ 1258us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:18.602422 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:18.602612 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:18.602978 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:18.602982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:18.603006 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:21:18.603518 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:18.603540 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.605002 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.605006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:18.608064 (+ 3058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:18.608083 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.609861 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.609866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:18.610126 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:18.610254 (+ 128us) client_negotiation.cc:769] Sending connection context
0929 20:21:18.610355 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 20:21:18.610655 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1096,"mutex_wait_us":297,"thread_start_us":958,"threads_started":1}
I20260929 20:21:18.611145 14696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:18.601268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:39173 (local address 127.27.185.66:51739)
0929 20:21:18.602433 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:18.602452 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:18.602659 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:18.603165 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:18.603169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:18.603197 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:21:18.603740 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:18.603753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.605122 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.605127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:18.608064 (+ 2937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:18.608083 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.609852 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.609859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:18.610128 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:18.610251 (+ 123us) client_negotiation.cc:769] Sending connection context
0929 20:21:18.610354 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 20:21:18.610616 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1054,"mutex_wait_us":282,"spinlock_wait_cycles":39296,"thread_start_us":658,"threads_started":1}
I20260929 20:21:18.612262 14355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065507805aae403da1ea23233ec27d67" candidate_uuid: "17f2fc9fd28647e7b2969e41cf2767de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cd131c34f0401a9cad6e0e2f0a2caf" is_pre_election: true
I20260929 20:21:18.612342 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065507805aae403da1ea23233ec27d67" candidate_uuid: "17f2fc9fd28647e7b2969e41cf2767de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f9a4f342414eb08754da901064a2b1" is_pre_election: true
I20260929 20:21:18.612840 14355 raft_consensus.cc:2454] T 065507805aae403da1ea23233ec27d67 P 92cd131c34f0401a9cad6e0e2f0a2caf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17f2fc9fd28647e7b2969e41cf2767de in term 0.
I20260929 20:21:18.612891 14624 raft_consensus.cc:2454] T 065507805aae403da1ea23233ec27d67 P c0f9a4f342414eb08754da901064a2b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17f2fc9fd28647e7b2969e41cf2767de in term 0.
I20260929 20:21:18.613585 14424 leader_election.cc:302] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17f2fc9fd28647e7b2969e41cf2767de, 92cd131c34f0401a9cad6e0e2f0a2caf, c0f9a4f342414eb08754da901064a2b1; no voters:
I20260929 20:21:18.614012 14695 raft_consensus.cc:2781] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:21:18.614166 14695 raft_consensus.cc:482] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:21:18.614269 14695 raft_consensus.cc:3037] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:18.616568 14695 raft_consensus.cc:504] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f9a4f342414eb08754da901064a2b1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40833 } } peers { permanent_uuid: "92cd131c34f0401a9cad6e0e2f0a2caf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 39173 } } peers { permanent_uuid: "17f2fc9fd28647e7b2969e41cf2767de" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 34783 } }
I20260929 20:21:18.617278 14695 leader_election.cc:288] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de [CANDIDATE]: Term 1 election: Requested vote from peers c0f9a4f342414eb08754da901064a2b1 (127.27.185.67:40833), 92cd131c34f0401a9cad6e0e2f0a2caf (127.27.185.65:39173)
I20260929 20:21:18.617734 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065507805aae403da1ea23233ec27d67" candidate_uuid: "17f2fc9fd28647e7b2969e41cf2767de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f9a4f342414eb08754da901064a2b1"
I20260929 20:21:18.617803 14355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065507805aae403da1ea23233ec27d67" candidate_uuid: "17f2fc9fd28647e7b2969e41cf2767de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cd131c34f0401a9cad6e0e2f0a2caf"
I20260929 20:21:18.617965 14624 raft_consensus.cc:3037] T 065507805aae403da1ea23233ec27d67 P c0f9a4f342414eb08754da901064a2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:18.617995 14355 raft_consensus.cc:3037] T 065507805aae403da1ea23233ec27d67 P 92cd131c34f0401a9cad6e0e2f0a2caf [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:18.621477 14355 raft_consensus.cc:2454] T 065507805aae403da1ea23233ec27d67 P 92cd131c34f0401a9cad6e0e2f0a2caf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17f2fc9fd28647e7b2969e41cf2767de in term 1.
I20260929 20:21:18.621477 14624 raft_consensus.cc:2454] T 065507805aae403da1ea23233ec27d67 P c0f9a4f342414eb08754da901064a2b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17f2fc9fd28647e7b2969e41cf2767de in term 1.
I20260929 20:21:18.621973 14424 leader_election.cc:302] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17f2fc9fd28647e7b2969e41cf2767de, 92cd131c34f0401a9cad6e0e2f0a2caf, c0f9a4f342414eb08754da901064a2b1; no voters:
I20260929 20:21:18.622268 14695 raft_consensus.cc:2781] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:21:18.622717 14695 raft_consensus.cc:686] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de [term 1 LEADER]: Becoming Leader. State: Replica: 17f2fc9fd28647e7b2969e41cf2767de, State: Running, Role: LEADER
I20260929 20:21:18.623260 14695 consensus_queue.cc:237] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f9a4f342414eb08754da901064a2b1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40833 } } peers { permanent_uuid: "92cd131c34f0401a9cad6e0e2f0a2caf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 39173 } } peers { permanent_uuid: "17f2fc9fd28647e7b2969e41cf2767de" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 34783 } }
I20260929 20:21:18.628352 14210 catalog_manager.cc:5495] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de reported cstate change: term changed from 0 to 1, leader changed from <none> to 17f2fc9fd28647e7b2969e41cf2767de (127.27.185.66). New cstate: current_term: 1 leader_uuid: "17f2fc9fd28647e7b2969e41cf2767de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f9a4f342414eb08754da901064a2b1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cd131c34f0401a9cad6e0e2f0a2caf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 39173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17f2fc9fd28647e7b2969e41cf2767de" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 34783 } health_report { overall_health: HEALTHY } } }
W20260929 20:21:18.644738 14402 tablet.cc:2369] T 065507805aae403da1ea23233ec27d67 P 92cd131c34f0401a9cad6e0e2f0a2caf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:21:18.661603 14686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:18.655424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42622 (local address 127.27.185.66:34783)
0929 20:21:18.655671 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:18.655676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:18.655699 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:21:18.655817 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:18.655821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:18.655897 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:18.656056 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:21:18.656470 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.657209 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.658625 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.659275 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.659429 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:18.660086 (+ 657us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:21:18.660870 (+ 784us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:21:18.661079 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:18.661230 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 20:21:18.661282 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 20:21:18.670562 14624 raft_consensus.cc:1261] T 065507805aae403da1ea23233ec27d67 P c0f9a4f342414eb08754da901064a2b1 [term 1 FOLLOWER]: Refusing update from remote peer 17f2fc9fd28647e7b2969e41cf2767de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:21:18.670782 14355 raft_consensus.cc:1261] T 065507805aae403da1ea23233ec27d67 P 92cd131c34f0401a9cad6e0e2f0a2caf [term 1 FOLLOWER]: Refusing update from remote peer 17f2fc9fd28647e7b2969e41cf2767de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:21:18.671842 14695 consensus_queue.cc:1035] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cd131c34f0401a9cad6e0e2f0a2caf" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 39173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:21:18.672192 14698 consensus_queue.cc:1035] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f9a4f342414eb08754da901064a2b1" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 40833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 20:21:18.673031 14536 tablet.cc:2369] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:21:18.683642 14670 tablet.cc:2369] T 065507805aae403da1ea23233ec27d67 P c0f9a4f342414eb08754da901064a2b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:21:18.688988 14703 mvcc.cc:205] Tried to move back new op lower bound from 7334761589424517120 to 7334761589245722624. Current Snapshot: MvccSnapshot[applied={T|T < 7334761589424517120}]
I20260929 20:21:18.699931 14210 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:21:18.700248 14210 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:21:18.704051 14210 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54dfd668b57f4814af9ae701b33e1f44: Sending DeleteTablet for 3 replicas of tablet 065507805aae403da1ea23233ec27d67
I20260929 20:21:18.705294 14604 tablet_service.cc:1514] Processing DeleteTablet for tablet 065507805aae403da1ea23233ec27d67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:21:18 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:53944
I20260929 20:21:18.705442 14335 tablet_service.cc:1514] Processing DeleteTablet for tablet 065507805aae403da1ea23233ec27d67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:21:18 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:44684
I20260929 20:21:18.705611 14470 tablet_service.cc:1514] Processing DeleteTablet for tablet 065507805aae403da1ea23233ec27d67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:21:18 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:42616
I20260929 20:21:18.708477 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 14272
I20260929 20:21:18.718017 14712 tablet_replica.cc:330] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de: stopping tablet replica
I20260929 20:21:18.718096 14711 tablet_replica.cc:330] T 065507805aae403da1ea23233ec27d67 P 92cd131c34f0401a9cad6e0e2f0a2caf: stopping tablet replica
I20260929 20:21:18.718637 14712 raft_consensus.cc:2229] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:21:18.718928 14711 raft_consensus.cc:2229] T 065507805aae403da1ea23233ec27d67 P 92cd131c34f0401a9cad6e0e2f0a2caf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:21:18.719516 14712 raft_consensus.cc:2258] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:21:18.719586 14711 raft_consensus.cc:2258] T 065507805aae403da1ea23233ec27d67 P 92cd131c34f0401a9cad6e0e2f0a2caf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:21:18.722075 14712 ts_tablet_manager.cc:1905] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:21:18.722414 14711 ts_tablet_manager.cc:1905] T 065507805aae403da1ea23233ec27d67 P 92cd131c34f0401a9cad6e0e2f0a2caf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:21:18.730196 14711 ts_tablet_manager.cc:1918] T 065507805aae403da1ea23233ec27d67 P 92cd131c34f0401a9cad6e0e2f0a2caf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:21:18.730508 14711 log.cc:1196] T 065507805aae403da1ea23233ec27d67 P 92cd131c34f0401a9cad6e0e2f0a2caf: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-0/wal/wals/065507805aae403da1ea23233ec27d67
I20260929 20:21:18.731155 14711 ts_tablet_manager.cc:1939] T 065507805aae403da1ea23233ec27d67 P 92cd131c34f0401a9cad6e0e2f0a2caf: Deleting consensus metadata
I20260929 20:21:18.731925 14710 tablet_replica.cc:330] T 065507805aae403da1ea23233ec27d67 P c0f9a4f342414eb08754da901064a2b1: stopping tablet replica
I20260929 20:21:18.732445 14710 raft_consensus.cc:2229] T 065507805aae403da1ea23233ec27d67 P c0f9a4f342414eb08754da901064a2b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:21:18.732911 14710 raft_consensus.cc:2258] T 065507805aae403da1ea23233ec27d67 P c0f9a4f342414eb08754da901064a2b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:21:18.733394 14196 catalog_manager.cc:4841] TS 92cd131c34f0401a9cad6e0e2f0a2caf (127.27.185.65:39173): tablet 065507805aae403da1ea23233ec27d67 (table test-table [id=0b040bf87f7b458cbc6f925b4c675bea]) successfully deleted
I20260929 20:21:18.733738 14712 ts_tablet_manager.cc:1918] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:21:18.733930 14712 log.cc:1196] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-1/wal/wals/065507805aae403da1ea23233ec27d67
I20260929 20:21:18.734390 14712 ts_tablet_manager.cc:1939] T 065507805aae403da1ea23233ec27d67 P 17f2fc9fd28647e7b2969e41cf2767de: Deleting consensus metadata
I20260929 20:21:18.735644 14710 ts_tablet_manager.cc:1905] T 065507805aae403da1ea23233ec27d67 P c0f9a4f342414eb08754da901064a2b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:21:18.736305 14197 catalog_manager.cc:4841] TS 17f2fc9fd28647e7b2969e41cf2767de (127.27.185.66:34783): tablet 065507805aae403da1ea23233ec27d67 (table test-table [id=0b040bf87f7b458cbc6f925b4c675bea]) successfully deleted
I20260929 20:21:18.742828 14710 ts_tablet_manager.cc:1918] T 065507805aae403da1ea23233ec27d67 P c0f9a4f342414eb08754da901064a2b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:21:18.743026 14710 log.cc:1196] T 065507805aae403da1ea23233ec27d67 P c0f9a4f342414eb08754da901064a2b1: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790712966917858-28389-0/minicluster-data/ts-2/wal/wals/065507805aae403da1ea23233ec27d67
I20260929 20:21:18.743630 14710 ts_tablet_manager.cc:1939] T 065507805aae403da1ea23233ec27d67 P c0f9a4f342414eb08754da901064a2b1: Deleting consensus metadata
I20260929 20:21:18.745759 14198 catalog_manager.cc:4841] TS c0f9a4f342414eb08754da901064a2b1 (127.27.185.67:40833): tablet 065507805aae403da1ea23233ec27d67 (table test-table [id=0b040bf87f7b458cbc6f925b4c675bea]) successfully deleted
Sep 29 20:21:18 dist-test-slave-4cx3 krb5kdc[14166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.65@KRBTEST.COM
I20260929 20:21:18.754554 14684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:18.710124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44698 (local address 127.27.185.65:39173)
0929 20:21:18.710361 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:18.710371 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:18.728979 (+ 18608us) server_negotiation.cc:407] Connection header received
0929 20:21:18.729508 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:18.729515 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:18.729600 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:18.729666 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:18.731624 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.732768 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.734706 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.735299 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.735489 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:18.748418 (+ 12929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:18.748448 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:18.748464 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:18.748522 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:18.751535 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:18.752337 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:18.752345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:18.752347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:18.752497 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:18.752902 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:18.752906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:18.752908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:18.753276 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:18.753599 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:18.753908 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 20:21:18.754135 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:21:18.755304 14396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:18.848491 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 14272
I20260929 20:21:18.867864 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 14407
Sep 29 20:21:18 dist-test-slave-4cx3 krb5kdc[14166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.66@KRBTEST.COM
I20260929 20:21:18.886817 14686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:18.869022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42638 (local address 127.27.185.66:34783)
0929 20:21:18.869237 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:18.869244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:18.869270 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:21:18.869355 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:18.869361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:18.869442 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:18.869550 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:18.870830 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.872143 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.873569 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:18.874282 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:18.874418 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:18.879239 (+ 4821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:18.879266 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:18.879288 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:18.879335 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:18.883873 (+ 4538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:18.884629 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:18.884639 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:18.884643 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:18.884814 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:18.885164 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:18.885168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:18.885172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:18.885659 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:18.886017 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:18.886165 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 20:21:18.886394 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:21:18.887683 14530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:19.001348 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 14407
I20260929 20:21:19.021109 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 14541
Sep 29 20:21:19 dist-test-slave-4cx3 krb5kdc[14166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.67@KRBTEST.COM
I20260929 20:21:19.038067 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:19.021935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53954 (local address 127.27.185.67:40833)
0929 20:21:19.022144 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:19.022152 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:19.022222 (+ 70us) server_negotiation.cc:407] Connection header received
0929 20:21:19.022501 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:19.022510 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:19.022589 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:19.022713 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:19.024484 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:19.025245 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:19.026571 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:19.027000 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:19.027159 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:19.032108 (+ 4949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:19.032125 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:19.032141 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:19.032182 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:19.034997 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:19.036003 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:19.036010 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:19.036013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:19.036197 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:19.036635 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:19.036639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:19.036641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:19.037026 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:19.037250 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:19.037630 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 20:21:19.037817 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:21:19.038851 14664 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:19.129814 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 14541
I20260929 20:21:19.146697 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 14179
I20260929 20:21:19.147897 14241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:19.260366 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 14179
2026-09-29T20:21:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4352 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T20:21:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:21:19Z Disabled control of system clock
I20260929 20:21:19.337446 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:41417
--webserver_interface=127.27.185.126
--webserver_port=0
--builtin_ntp_servers=127.27.185.84:41087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:41417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 20:21:19.678169 14721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:19.678489 14721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:19.678639 14721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:19.686352 14721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:21:19.686470 14721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:19.686548 14721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:19.686656 14721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:21:19.686753 14721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:21:19.696606 14721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:41087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:41417
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:41417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:19.697968 14721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:19.699889 14721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:19.712023 14726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:19.712214 14729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:19.712214 14727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:19.713201 14721 server_base.cc:1034] running on GCE node
I20260929 20:21:19.713972 14721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:19.715421 14721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:19.716648 14721 hybrid_clock.cc:648] HybridClock initialized: now 1790713279716603 us; error 37 us; skew 500 ppm
I20260929 20:21:19.717028 14721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:21:19.719372 14721 webserver.cc:460] Webserver started at http://127.27.185.126:33653/ using document root <none> and password file <none>
I20260929 20:21:19.720139 14721 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:19.720255 14721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:19.720606 14721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:19.723589 14721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "7d39f74b12904d6f9eaaf93a75541b40"
format_stamp: "Formatted at 2026-09-29 20:21:19 on dist-test-slave-4cx3"
server_key: "5ad845320697e30a606a4d18bc95d87b"
server_key_iv: "2f819584fe2a8716173ed4aa641675d9"
server_key_version: "encryptionkey@0"
I20260929 20:21:19.724429 14721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "7d39f74b12904d6f9eaaf93a75541b40"
format_stamp: "Formatted at 2026-09-29 20:21:19 on dist-test-slave-4cx3"
server_key: "5ad845320697e30a606a4d18bc95d87b"
server_key_iv: "2f819584fe2a8716173ed4aa641675d9"
server_key_version: "encryptionkey@0"
I20260929 20:21:19.729688 14721 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:21:19.733294 14735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:19.734722 14721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:21:19.734936 14721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "7d39f74b12904d6f9eaaf93a75541b40"
format_stamp: "Formatted at 2026-09-29 20:21:19 on dist-test-slave-4cx3"
server_key: "5ad845320697e30a606a4d18bc95d87b"
server_key_iv: "2f819584fe2a8716173ed4aa641675d9"
server_key_version: "encryptionkey@0"
I20260929 20:21:19.735179 14721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:19.750138 14721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:19.751157 14721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:19.751521 14721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:19.770437 14721 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:41417
I20260929 20:21:19.770462 14786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:41417 every 8 connection(s)
I20260929 20:21:19.771929 14721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:21:19.775878 14787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:19.782088 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 14721
I20260929 20:21:19.782289 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:21:19.782856 28389 external_mini_cluster.cc:1428] Setting key 70f26f182cbdc9204a40673296bff251
I20260929 20:21:19.787174 14787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d39f74b12904d6f9eaaf93a75541b40: Bootstrap starting.
I20260929 20:21:19.790704 14787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d39f74b12904d6f9eaaf93a75541b40: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:19.791805 14787 log.cc:824] T 00000000000000000000000000000000 P 7d39f74b12904d6f9eaaf93a75541b40: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:19.794669 14787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d39f74b12904d6f9eaaf93a75541b40: No bootstrap required, opened a new log
I20260929 20:21:19.797005 14790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:19.784899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46896 (local address 127.27.185.126:41417)
0929 20:21:19.785976 (+ 1077us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:19.785989 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:19.786034 (+ 45us) server_negotiation.cc:407] Connection header received
0929 20:21:19.787934 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:19.787958 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:19.788459 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:19.788820 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:19.789478 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:19.790853 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:19.792391 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:19.793122 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:19.793329 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:19.794523 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:19.794531 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:19.794546 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:19.795176 (+ 630us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:19.795255 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:19.795276 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:19.795426 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:19.795656 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 20:21:19.795978 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":381,"threads_started":1}
I20260929 20:21:19.803151 14787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d39f74b12904d6f9eaaf93a75541b40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d39f74b12904d6f9eaaf93a75541b40" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41417 } }
I20260929 20:21:19.803601 14787 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d39f74b12904d6f9eaaf93a75541b40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:19.803709 14787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d39f74b12904d6f9eaaf93a75541b40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d39f74b12904d6f9eaaf93a75541b40, State: Initialized, Role: FOLLOWER
I20260929 20:21:19.804330 14787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d39f74b12904d6f9eaaf93a75541b40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d39f74b12904d6f9eaaf93a75541b40" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41417 } }
I20260929 20:21:19.804565 14787 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d39f74b12904d6f9eaaf93a75541b40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:21:19.804718 14787 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d39f74b12904d6f9eaaf93a75541b40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:21:19.804898 14787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d39f74b12904d6f9eaaf93a75541b40 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:19.807374 14787 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d39f74b12904d6f9eaaf93a75541b40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d39f74b12904d6f9eaaf93a75541b40" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41417 } }
I20260929 20:21:19.807933 14787 leader_election.cc:302] T 00000000000000000000000000000000 P 7d39f74b12904d6f9eaaf93a75541b40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d39f74b12904d6f9eaaf93a75541b40; no voters:
I20260929 20:21:19.808509 14787 leader_election.cc:288] T 00000000000000000000000000000000 P 7d39f74b12904d6f9eaaf93a75541b40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:21:19.808670 14792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d39f74b12904d6f9eaaf93a75541b40 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:21:19.809412 14792 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d39f74b12904d6f9eaaf93a75541b40 [term 1 LEADER]: Becoming Leader. State: Replica: 7d39f74b12904d6f9eaaf93a75541b40, State: Running, Role: LEADER
I20260929 20:21:19.809937 14792 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d39f74b12904d6f9eaaf93a75541b40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d39f74b12904d6f9eaaf93a75541b40" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41417 } }
I20260929 20:21:19.810554 14787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d39f74b12904d6f9eaaf93a75541b40 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:21:19.813160 14793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d39f74b12904d6f9eaaf93a75541b40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d39f74b12904d6f9eaaf93a75541b40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d39f74b12904d6f9eaaf93a75541b40" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41417 } } }
I20260929 20:21:19.813516 14793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d39f74b12904d6f9eaaf93a75541b40 [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:19.814428 14801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:21:19.812961 14794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d39f74b12904d6f9eaaf93a75541b40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d39f74b12904d6f9eaaf93a75541b40. Latest consensus state: current_term: 1 leader_uuid: "7d39f74b12904d6f9eaaf93a75541b40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d39f74b12904d6f9eaaf93a75541b40" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41417 } } }
I20260929 20:21:19.815137 14794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d39f74b12904d6f9eaaf93a75541b40 [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:19.820722 14801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:21:19.830452 14801 catalog_manager.cc:1347] Generated new cluster ID: 476fc6c08e80496aaad0a39e8aab7d2b
I20260929 20:21:19.830593 14801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:21:19.849138 14801 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:21:19.851114 14801 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:21:19.858939 14801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d39f74b12904d6f9eaaf93a75541b40: Generated new TSK 0
I20260929 20:21:19.860105 14801 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:21:19.871906 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:0
--local_ip_for_outbound_sockets=127.27.185.65
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:41417
--builtin_ntp_servers=127.27.185.84:41087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 20:21:20.215688 14811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:20.215982 14811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:20.216157 14811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:20.223555 14811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:20.223665 14811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:20.223841 14811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:21:20.233527 14811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:41087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:41417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:20.234891 14811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:20.236815 14811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:20.250746 14816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:20.250782 14819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:20.250762 14817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:20.251824 14811 server_base.cc:1034] running on GCE node
I20260929 20:21:20.252473 14811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:20.253511 14811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:20.254730 14811 hybrid_clock.cc:648] HybridClock initialized: now 1790713280254656 us; error 70 us; skew 500 ppm
I20260929 20:21:20.255062 14811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:21:20.257488 14811 webserver.cc:460] Webserver started at http://127.27.185.65:41055/ using document root <none> and password file <none>
I20260929 20:21:20.258219 14811 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:20.258358 14811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:20.258692 14811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:20.261857 14811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-0/data/instance:
uuid: "6001c4a29f36466390d35a1e3133f756"
format_stamp: "Formatted at 2026-09-29 20:21:20 on dist-test-slave-4cx3"
server_key: "31c7cfc3bf08917817d756f1b5f8ef69"
server_key_iv: "60dc056a090b79b035216b04b626c006"
server_key_version: "encryptionkey@0"
I20260929 20:21:20.262715 14811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance:
uuid: "6001c4a29f36466390d35a1e3133f756"
format_stamp: "Formatted at 2026-09-29 20:21:20 on dist-test-slave-4cx3"
server_key: "31c7cfc3bf08917817d756f1b5f8ef69"
server_key_iv: "60dc056a090b79b035216b04b626c006"
server_key_version: "encryptionkey@0"
I20260929 20:21:20.268402 14811 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260929 20:21:20.272214 14825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:20.273658 14811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:21:20.273844 14811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "6001c4a29f36466390d35a1e3133f756"
format_stamp: "Formatted at 2026-09-29 20:21:20 on dist-test-slave-4cx3"
server_key: "31c7cfc3bf08917817d756f1b5f8ef69"
server_key_iv: "60dc056a090b79b035216b04b626c006"
server_key_version: "encryptionkey@0"
I20260929 20:21:20.274073 14811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:20.302798 14811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:20.303821 14811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:20.304158 14811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:20.305173 14811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:21:20.306682 14811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:21:20.306807 14811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:20.306946 14811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:21:20.307049 14811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:20.341192 14811 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:44523
I20260929 20:21:20.341253 14937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:44523 every 8 connection(s)
I20260929 20:21:20.342556 14811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:21:20.347874 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 14811
I20260929 20:21:20.348081 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance
I20260929 20:21:20.348636 28389 external_mini_cluster.cc:1428] Setting key 1bede5e99522bb523dfd7cdb9fd2c543
I20260929 20:21:20.351821 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:0
--local_ip_for_outbound_sockets=127.27.185.66
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:41417
--builtin_ntp_servers=127.27.185.84:41087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 20:21:20.357966 14940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:20.345287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:48079 (local address 127.27.185.126:41417)
0929 20:21:20.345733 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:20.345739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:20.347101 (+ 1362us) server_negotiation.cc:407] Connection header received
0929 20:21:20.349655 (+ 2554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:20.349659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:20.349768 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:20.349888 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:20.351270 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:20.352230 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:20.354822 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:20.355229 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:20.355319 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:20.357102 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:20.357107 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:20.357110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:20.357131 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:20.357177 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:20.357189 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:20.357255 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:20.357543 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 20:21:20.357626 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":205,"threads_started":1}
I20260929 20:21:20.359603 14941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:20.345996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41417 (local address 127.27.185.65:48079)
0929 20:21:20.346929 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:20.346976 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:20.349302 (+ 2326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:20.350100 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:20.350109 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:20.350203 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:21:20.351083 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:20.351116 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:20.352382 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:20.352390 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:20.354675 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:20.354693 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:20.355491 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:20.355501 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:20.355861 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:20.356698 (+ 837us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:21:20.356703 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:21:20.356767 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:21:20.356881 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:21:20.357285 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:21:20.357300 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:21:20.357311 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 20:21:20.357532 (+ 221us) client_negotiation.cc:240] Negotiation successful
0929 20:21:20.358014 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":277,"threads_started":1}
I20260929 20:21:20.361359 14938 heartbeater.cc:347] Connected to a master server at 127.27.185.126:41417
I20260929 20:21:20.361756 14938 heartbeater.cc:464] Registering TS with master...
I20260929 20:21:20.362717 14938 heartbeater.cc:511] Master 127.27.185.126:41417 requested a full tablet report, sending...
I20260929 20:21:20.364998 14752 ts_manager.cc:194] Registered new tserver with Master: 6001c4a29f36466390d35a1e3133f756 (127.27.185.65:44523)
I20260929 20:21:20.366518 14752 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.185.65:48079
W20260929 20:21:20.689464 14942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:20.689805 14942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:20.689972 14942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:20.697248 14942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:20.697397 14942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:20.697593 14942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:21:20.706771 14942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:41087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:41417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:20.708176 14942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:20.709991 14942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:20.723419 14948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:20.723608 14947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:20.723658 14950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:20.724314 14942 server_base.cc:1034] running on GCE node
I20260929 20:21:20.724926 14942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:20.725869 14942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:20.727098 14942 hybrid_clock.cc:648] HybridClock initialized: now 1790713280727049 us; error 62 us; skew 500 ppm
I20260929 20:21:20.727478 14942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:21:20.729718 14942 webserver.cc:460] Webserver started at http://127.27.185.66:46449/ using document root <none> and password file <none>
I20260929 20:21:20.730422 14942 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:20.730561 14942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:20.730880 14942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:20.733832 14942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-1/data/instance:
uuid: "79382a0ed9294a32aa664364947c8a06"
format_stamp: "Formatted at 2026-09-29 20:21:20 on dist-test-slave-4cx3"
server_key: "a703cfe061d127f1231bae11f8eae7f1"
server_key_iv: "e9d6fcfefaac8f136cc93cb02f95778b"
server_key_version: "encryptionkey@0"
I20260929 20:21:20.734644 14942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance:
uuid: "79382a0ed9294a32aa664364947c8a06"
format_stamp: "Formatted at 2026-09-29 20:21:20 on dist-test-slave-4cx3"
server_key: "a703cfe061d127f1231bae11f8eae7f1"
server_key_iv: "e9d6fcfefaac8f136cc93cb02f95778b"
server_key_version: "encryptionkey@0"
I20260929 20:21:20.739977 14942 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:21:20.743536 14956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:20.744900 14942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:21:20.745080 14942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "79382a0ed9294a32aa664364947c8a06"
format_stamp: "Formatted at 2026-09-29 20:21:20 on dist-test-slave-4cx3"
server_key: "a703cfe061d127f1231bae11f8eae7f1"
server_key_iv: "e9d6fcfefaac8f136cc93cb02f95778b"
server_key_version: "encryptionkey@0"
I20260929 20:21:20.745292 14942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:20.768537 14942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:20.769443 14942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:20.769779 14942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:20.770807 14942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:21:20.772385 14942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:21:20.772508 14942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:20.772653 14942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:21:20.772742 14942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:20.806928 14942 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:35243
I20260929 20:21:20.807017 15068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:35243 every 8 connection(s)
I20260929 20:21:20.808310 14942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:21:20.810536 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 14942
I20260929 20:21:20.810717 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance
I20260929 20:21:20.811293 28389 external_mini_cluster.cc:1428] Setting key 8d29e5ca4bfb0ddb0931843bd2c0cddb
I20260929 20:21:20.814441 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:0
--local_ip_for_outbound_sockets=127.27.185.67
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:41417
--builtin_ntp_servers=127.27.185.84:41087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 20:21:20.821611 14940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:20.811215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:47679 (local address 127.27.185.126:41417)
0929 20:21:20.811472 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:20.811478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:20.813092 (+ 1614us) server_negotiation.cc:407] Connection header received
0929 20:21:20.815192 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:20.815197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:20.815284 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:20.815420 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:20.816515 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:20.817242 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:20.818816 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:20.819229 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:20.819366 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:20.820788 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:20.820794 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:20.820799 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:20.820828 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:20.820859 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:20.820870 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:20.820966 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:20.821204 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 20:21:20.821303 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:21:20.822858 15071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:20.811809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41417 (local address 127.27.185.66:47679)
0929 20:21:20.812922 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:20.812977 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:20.814921 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:20.815665 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:20.815674 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:20.815769 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:21:20.816344 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:20.816361 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:20.817408 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:20.817420 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:20.818664 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:20.818675 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:20.819478 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:20.819486 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:20.819733 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:20.820452 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:21:20.820455 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:21:20.820502 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:21:20.820594 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:21:20.820964 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:21:20.820977 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:21:20.820988 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 20:21:20.821172 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 20:21:20.821595 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":326,"threads_started":1}
I20260929 20:21:20.824354 15069 heartbeater.cc:347] Connected to a master server at 127.27.185.126:41417
I20260929 20:21:20.824692 15069 heartbeater.cc:464] Registering TS with master...
I20260929 20:21:20.825599 15069 heartbeater.cc:511] Master 127.27.185.126:41417 requested a full tablet report, sending...
I20260929 20:21:20.827194 14752 ts_manager.cc:194] Registered new tserver with Master: 79382a0ed9294a32aa664364947c8a06 (127.27.185.66:35243)
I20260929 20:21:20.828459 14752 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.185.66:47679
W20260929 20:21:21.149374 15072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:21.149713 15072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:21.149861 15072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:21.157455 15072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:21.157572 15072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:21.157748 15072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:21:21.167062 15072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:41087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:41417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:21.168438 15072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:21.170354 15072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:21.184592 15077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:21.184561 15080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:21.184641 15078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:21.189003 15072 server_base.cc:1034] running on GCE node
I20260929 20:21:21.189693 15072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:21.190775 15072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:21.192058 15072 hybrid_clock.cc:648] HybridClock initialized: now 1790713281191984 us; error 63 us; skew 500 ppm
I20260929 20:21:21.192435 15072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:21:21.194834 15072 webserver.cc:460] Webserver started at http://127.27.185.67:46405/ using document root <none> and password file <none>
I20260929 20:21:21.195647 15072 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:21.195761 15072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:21.196098 15072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:21.199088 15072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-2/data/instance:
uuid: "e41bbe7b55d64a498c169a52f8abaf1a"
format_stamp: "Formatted at 2026-09-29 20:21:21 on dist-test-slave-4cx3"
server_key: "de0614ece8ddacdfbc6a725c76df0448"
server_key_iv: "1a68554034202f9b77865f525449dd71"
server_key_version: "encryptionkey@0"
I20260929 20:21:21.199966 15072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance:
uuid: "e41bbe7b55d64a498c169a52f8abaf1a"
format_stamp: "Formatted at 2026-09-29 20:21:21 on dist-test-slave-4cx3"
server_key: "de0614ece8ddacdfbc6a725c76df0448"
server_key_iv: "1a68554034202f9b77865f525449dd71"
server_key_version: "encryptionkey@0"
I20260929 20:21:21.205315 15072 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:21:21.209020 15086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:21.210433 15072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:21:21.210611 15072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "e41bbe7b55d64a498c169a52f8abaf1a"
format_stamp: "Formatted at 2026-09-29 20:21:21 on dist-test-slave-4cx3"
server_key: "de0614ece8ddacdfbc6a725c76df0448"
server_key_iv: "1a68554034202f9b77865f525449dd71"
server_key_version: "encryptionkey@0"
I20260929 20:21:21.210834 15072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:21.233174 15072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:21.234069 15072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:21.234396 15072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:21.235500 15072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:21:21.237066 15072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:21:21.237190 15072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:21.237329 15072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:21:21.237416 15072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:21.272130 15072 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:35945
I20260929 20:21:21.272233 15198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:35945 every 8 connection(s)
I20260929 20:21:21.273638 15072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:21:21.282507 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 15072
I20260929 20:21:21.282694 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance
I20260929 20:21:21.283172 28389 external_mini_cluster.cc:1428] Setting key f42c3ec6c2f786f5964058765cf52e62
I20260929 20:21:21.287659 14940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:21.276407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:34291 (local address 127.27.185.126:41417)
0929 20:21:21.276615 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:21.276620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:21.278141 (+ 1521us) server_negotiation.cc:407] Connection header received
0929 20:21:21.280108 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:21.280113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:21.280209 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:21.280322 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:21.281433 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.282261 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.284364 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.284988 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.285141 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:21.286820 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:21.286826 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:21.286830 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:21.286852 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:21.286899 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:21.286915 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:21.287000 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:21.287203 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 20:21:21.287283 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 20:21:21.288748 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:21.276956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41417 (local address 127.27.185.67:34291)
0929 20:21:21.277979 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:21.278024 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:21.279805 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:21.280572 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:21.280581 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:21.280680 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:21:21.281256 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:21.281273 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.282461 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.282465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:21.284209 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:21.284224 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.285256 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.285267 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:21.285607 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:21.286491 (+ 884us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:21:21.286494 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:21:21.286537 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:21:21.286623 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:21:21.287016 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:21:21.287024 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:21:21.287030 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 20:21:21.287174 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 20:21:21.287652 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":318,"threads_started":1}
I20260929 20:21:21.290103 15199 heartbeater.cc:347] Connected to a master server at 127.27.185.126:41417
I20260929 20:21:21.290412 15199 heartbeater.cc:464] Registering TS with master...
I20260929 20:21:21.291172 15199 heartbeater.cc:511] Master 127.27.185.126:41417 requested a full tablet report, sending...
I20260929 20:21:21.292606 14752 ts_manager.cc:194] Registered new tserver with Master: e41bbe7b55d64a498c169a52f8abaf1a (127.27.185.67:35945)
I20260929 20:21:21.293426 14752 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.185.67:34291
I20260929 20:21:21.298653 28389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:21:21.308808 14940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:21.302782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46912 (local address 127.27.185.126:41417)
0929 20:21:21.302962 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:21.302967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:21.303524 (+ 557us) server_negotiation.cc:407] Connection header received
0929 20:21:21.303673 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:21.303677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:21.303740 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:21.303825 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:21.304313 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.305048 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.306452 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.306868 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.306970 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:21.308154 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:21.308159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:21.308162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:21.308181 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:21.308214 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:21.308225 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:21.308337 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:21.308518 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 20:21:21.308564 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:21:21.313002 14752 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:21:21.319967 14752 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:21:21.351019 15213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:21.332632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:35243 (local address 127.0.0.1:44236)
0929 20:21:21.333580 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:21.333641 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:21.333866 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:21.336241 (+ 2375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:21.336250 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:21.336294 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:21:21.338462 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:21.338485 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.342725 (+ 4240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.342734 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:21.346777 (+ 4043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:21.346800 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.349859 (+ 3059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.349869 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:21.350206 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:21.350288 (+ 82us) client_negotiation.cc:769] Sending connection context
0929 20:21:21.350417 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 20:21:21.350625 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"spinlock_wait_cycles":16640,"thread_start_us":415,"threads_started":1}
I20260929 20:21:21.352644 15214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:21.333246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44236 (local address 127.27.185.66:35243)
0929 20:21:21.335081 (+ 1835us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:21.335094 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:21.335133 (+ 39us) server_negotiation.cc:407] Connection header received
0929 20:21:21.335255 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:21.335277 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:21.335637 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:21.335881 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:21:21.338751 (+ 2870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.342530 (+ 3779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.347105 (+ 4575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.349566 (+ 2461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.351688 (+ 2122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:21.351982 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:21.352065 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 20:21:21.352266 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1657,"thread_start_us":1545,"threads_started":1}
I20260929 20:21:21.357023 15215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:21.334586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:35945 (local address 127.0.0.1:43118)
0929 20:21:21.345565 (+ 10979us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:21.345588 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:21.345716 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:21.348674 (+ 2958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:21.348679 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:21.348711 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:21:21.349162 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:21.349176 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.351873 (+ 2697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.351879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:21.354038 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:21.354055 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.356283 (+ 2228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.356288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:21.356406 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:21.356429 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:21:21.356548 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 20:21:21.356687 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10838,"thread_start_us":11218,"threads_started":1}
I20260929 20:21:21.356611 15004 tablet_service.cc:1467] Processing CreateTablet for tablet 7bca90a3e2a64e8189570793805613e6 (DEFAULT_TABLE table=test-table [id=7505efe4851b45fb883e2e5a84621efd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:21:21.357247 15216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:21.334944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43118 (local address 127.27.185.67:35945)
0929 20:21:21.347664 (+ 12720us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:21.347674 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:21.347714 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:21:21.347838 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:21.347846 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:21.348132 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:21.348336 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:21:21.349599 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.351646 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.354274 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.355802 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.355966 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:21.356179 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:21.356597 (+ 418us) server_negotiation.cc:299] Negotiation successful
0929 20:21:21.356765 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12516,"thread_start_us":14356,"threads_started":1}
I20260929 20:21:21.358330 15004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bca90a3e2a64e8189570793805613e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:21.359562 15212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:21.332656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:44523 (local address 127.0.0.1:54420)
0929 20:21:21.340906 (+ 8250us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:21.340930 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:21.341051 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:21.351319 (+ 10268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:21.351324 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:21.351354 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:21:21.351722 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:21.351736 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.353340 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.353348 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:21.355505 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:21.355522 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.358835 (+ 3313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.358839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:21.358961 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:21.358984 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:21:21.359091 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 20:21:21.359231 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1758,"thread_start_us":1563,"threads_started":1}
I20260929 20:21:21.359750 15211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:21.332654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54420 (local address 127.27.185.65:44523)
0929 20:21:21.350694 (+ 18040us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:21.350702 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:21.350732 (+ 30us) server_negotiation.cc:407] Connection header received
0929 20:21:21.350824 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:21.350832 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:21.351087 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:21.351928 (+ 841us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:21:21.352106 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.353196 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.355839 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.358288 (+ 2449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.358437 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:21.358719 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:21.359156 (+ 437us) server_negotiation.cc:299] Negotiation successful
0929 20:21:21.359332 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17867,"thread_start_us":18659,"threads_started":1}
I20260929 20:21:21.361373 15134 tablet_service.cc:1467] Processing CreateTablet for tablet 7bca90a3e2a64e8189570793805613e6 (DEFAULT_TABLE table=test-table [id=7505efe4851b45fb883e2e5a84621efd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:21:21.362946 15134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bca90a3e2a64e8189570793805613e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:21.362807 14873 tablet_service.cc:1467] Processing CreateTablet for tablet 7bca90a3e2a64e8189570793805613e6 (DEFAULT_TABLE table=test-table [id=7505efe4851b45fb883e2e5a84621efd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:21:21.364022 14873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bca90a3e2a64e8189570793805613e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:21.369829 14938 heartbeater.cc:503] Master 127.27.185.126:41417 was elected leader, sending a full tablet report...
I20260929 20:21:21.373621 15217 tablet_bootstrap.cc:492] T 7bca90a3e2a64e8189570793805613e6 P 79382a0ed9294a32aa664364947c8a06: Bootstrap starting.
I20260929 20:21:21.376058 15218 tablet_bootstrap.cc:492] T 7bca90a3e2a64e8189570793805613e6 P 6001c4a29f36466390d35a1e3133f756: Bootstrap starting.
I20260929 20:21:21.378012 15217 tablet_bootstrap.cc:654] T 7bca90a3e2a64e8189570793805613e6 P 79382a0ed9294a32aa664364947c8a06: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:21.379438 15217 log.cc:824] T 7bca90a3e2a64e8189570793805613e6 P 79382a0ed9294a32aa664364947c8a06: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:21.379890 15218 tablet_bootstrap.cc:654] T 7bca90a3e2a64e8189570793805613e6 P 6001c4a29f36466390d35a1e3133f756: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:21.381294 15218 log.cc:824] T 7bca90a3e2a64e8189570793805613e6 P 6001c4a29f36466390d35a1e3133f756: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:21.382005 15220 tablet_bootstrap.cc:492] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a: Bootstrap starting.
I20260929 20:21:21.382779 15217 tablet_bootstrap.cc:492] T 7bca90a3e2a64e8189570793805613e6 P 79382a0ed9294a32aa664364947c8a06: No bootstrap required, opened a new log
I20260929 20:21:21.383069 15217 ts_tablet_manager.cc:1397] T 7bca90a3e2a64e8189570793805613e6 P 79382a0ed9294a32aa664364947c8a06: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260929 20:21:21.384472 15218 tablet_bootstrap.cc:492] T 7bca90a3e2a64e8189570793805613e6 P 6001c4a29f36466390d35a1e3133f756: No bootstrap required, opened a new log
I20260929 20:21:21.384784 15218 ts_tablet_manager.cc:1397] T 7bca90a3e2a64e8189570793805613e6 P 6001c4a29f36466390d35a1e3133f756: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260929 20:21:21.385636 15220 tablet_bootstrap.cc:654] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:21.386925 15220 log.cc:824] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:21.390324 15220 tablet_bootstrap.cc:492] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a: No bootstrap required, opened a new log
I20260929 20:21:21.390640 15220 ts_tablet_manager.cc:1397] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260929 20:21:21.391484 15217 raft_consensus.cc:348] T 7bca90a3e2a64e8189570793805613e6 P 79382a0ed9294a32aa664364947c8a06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6001c4a29f36466390d35a1e3133f756" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 44523 } } peers { permanent_uuid: "79382a0ed9294a32aa664364947c8a06" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 35243 } } peers { permanent_uuid: "e41bbe7b55d64a498c169a52f8abaf1a" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 35945 } }
I20260929 20:21:21.391884 15217 raft_consensus.cc:374] T 7bca90a3e2a64e8189570793805613e6 P 79382a0ed9294a32aa664364947c8a06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:21.391986 15217 raft_consensus.cc:729] T 7bca90a3e2a64e8189570793805613e6 P 79382a0ed9294a32aa664364947c8a06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79382a0ed9294a32aa664364947c8a06, State: Initialized, Role: FOLLOWER
I20260929 20:21:21.392642 15217 consensus_queue.cc:260] T 7bca90a3e2a64e8189570793805613e6 P 79382a0ed9294a32aa664364947c8a06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6001c4a29f36466390d35a1e3133f756" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 44523 } } peers { permanent_uuid: "79382a0ed9294a32aa664364947c8a06" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 35243 } } peers { permanent_uuid: "e41bbe7b55d64a498c169a52f8abaf1a" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 35945 } }
I20260929 20:21:21.393822 15069 heartbeater.cc:503] Master 127.27.185.126:41417 was elected leader, sending a full tablet report...
I20260929 20:21:21.394371 15217 ts_tablet_manager.cc:1428] T 7bca90a3e2a64e8189570793805613e6 P 79382a0ed9294a32aa664364947c8a06: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260929 20:21:21.397140 15218 raft_consensus.cc:348] T 7bca90a3e2a64e8189570793805613e6 P 6001c4a29f36466390d35a1e3133f756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6001c4a29f36466390d35a1e3133f756" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 44523 } } peers { permanent_uuid: "79382a0ed9294a32aa664364947c8a06" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 35243 } } peers { permanent_uuid: "e41bbe7b55d64a498c169a52f8abaf1a" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 35945 } }
I20260929 20:21:21.397624 15218 raft_consensus.cc:374] T 7bca90a3e2a64e8189570793805613e6 P 6001c4a29f36466390d35a1e3133f756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:21.397723 15218 raft_consensus.cc:729] T 7bca90a3e2a64e8189570793805613e6 P 6001c4a29f36466390d35a1e3133f756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6001c4a29f36466390d35a1e3133f756, State: Initialized, Role: FOLLOWER
I20260929 20:21:21.398450 15218 consensus_queue.cc:260] T 7bca90a3e2a64e8189570793805613e6 P 6001c4a29f36466390d35a1e3133f756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6001c4a29f36466390d35a1e3133f756" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 44523 } } peers { permanent_uuid: "79382a0ed9294a32aa664364947c8a06" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 35243 } } peers { permanent_uuid: "e41bbe7b55d64a498c169a52f8abaf1a" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 35945 } }
I20260929 20:21:21.400543 15218 ts_tablet_manager.cc:1428] T 7bca90a3e2a64e8189570793805613e6 P 6001c4a29f36466390d35a1e3133f756: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260929 20:21:21.401448 15220 raft_consensus.cc:348] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6001c4a29f36466390d35a1e3133f756" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 44523 } } peers { permanent_uuid: "79382a0ed9294a32aa664364947c8a06" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 35243 } } peers { permanent_uuid: "e41bbe7b55d64a498c169a52f8abaf1a" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 35945 } }
I20260929 20:21:21.401968 15220 raft_consensus.cc:374] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:21.402076 15220 raft_consensus.cc:729] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e41bbe7b55d64a498c169a52f8abaf1a, State: Initialized, Role: FOLLOWER
I20260929 20:21:21.402678 15220 consensus_queue.cc:260] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6001c4a29f36466390d35a1e3133f756" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 44523 } } peers { permanent_uuid: "79382a0ed9294a32aa664364947c8a06" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 35243 } } peers { permanent_uuid: "e41bbe7b55d64a498c169a52f8abaf1a" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 35945 } }
I20260929 20:21:21.403733 15199 heartbeater.cc:503] Master 127.27.185.126:41417 was elected leader, sending a full tablet report...
I20260929 20:21:21.404510 15220 ts_tablet_manager.cc:1428] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260929 20:21:21.499293 15225 raft_consensus.cc:482] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:21:21.499667 15225 raft_consensus.cc:504] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6001c4a29f36466390d35a1e3133f756" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 44523 } } peers { permanent_uuid: "79382a0ed9294a32aa664364947c8a06" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 35243 } } peers { permanent_uuid: "e41bbe7b55d64a498c169a52f8abaf1a" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 35945 } }
I20260929 20:21:21.501638 15225 leader_election.cc:288] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6001c4a29f36466390d35a1e3133f756 (127.27.185.65:44523), 79382a0ed9294a32aa664364947c8a06 (127.27.185.66:35243)
I20260929 20:21:21.507162 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:21.501845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:44523 (local address 127.27.185.67:36313)
0929 20:21:21.502143 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:21.502163 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:21.502303 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:21.502750 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:21.502756 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:21.502791 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:21:21.502970 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:21.502977 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.504035 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.504041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:21.505334 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:21.505345 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.506422 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.506428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:21.506610 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:21.506638 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 20:21:21.506736 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 20:21:21.506872 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260929 20:21:21.507151 15211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:21.501987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:36313 (local address 127.27.185.65:44523)
0929 20:21:21.502195 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:21.502203 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:21.502225 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:21:21.502485 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:21.502489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:21.502558 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:21.502661 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:21:21.503119 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.503902 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.505485 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.506259 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.506434 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:21.506507 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:21.506773 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 20:21:21.506904 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 20:21:21.508108 15226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:21.502138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:35243 (local address 127.27.185.67:47131)
0929 20:21:21.502914 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:21.502936 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:21.503107 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:21.503464 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:21.503469 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:21.503491 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:21:21.503794 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:21.503802 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.504761 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.504766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:21.506103 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:21.506114 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.507504 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.507511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:21.507680 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:21.507720 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 20:21:21.507818 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 20:21:21.507911 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":456,"threads_started":1}
I20260929 20:21:21.508276 15214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:21.502112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:47131 (local address 127.27.185.66:35243)
0929 20:21:21.502322 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:21.502330 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:21.503037 (+ 707us) server_negotiation.cc:407] Connection header received
0929 20:21:21.503265 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:21.503269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:21.503328 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:21.503446 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:21:21.503941 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.504612 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.506282 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.507184 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.507403 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:21.507467 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:21.507865 (+ 398us) server_negotiation.cc:299] Negotiation successful
0929 20:21:21.508006 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:21:21.508442 14893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bca90a3e2a64e8189570793805613e6" candidate_uuid: "e41bbe7b55d64a498c169a52f8abaf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6001c4a29f36466390d35a1e3133f756" is_pre_election: true
I20260929 20:21:21.509116 14893 raft_consensus.cc:2454] T 7bca90a3e2a64e8189570793805613e6 P 6001c4a29f36466390d35a1e3133f756 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e41bbe7b55d64a498c169a52f8abaf1a in term 0.
I20260929 20:21:21.509155 15024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bca90a3e2a64e8189570793805613e6" candidate_uuid: "e41bbe7b55d64a498c169a52f8abaf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79382a0ed9294a32aa664364947c8a06" is_pre_election: true
I20260929 20:21:21.509732 15024 raft_consensus.cc:2454] T 7bca90a3e2a64e8189570793805613e6 P 79382a0ed9294a32aa664364947c8a06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e41bbe7b55d64a498c169a52f8abaf1a in term 0.
I20260929 20:21:21.509853 15088 leader_election.cc:302] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6001c4a29f36466390d35a1e3133f756, e41bbe7b55d64a498c169a52f8abaf1a; no voters:
I20260929 20:21:21.510305 15225 raft_consensus.cc:2781] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:21:21.510479 15225 raft_consensus.cc:482] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:21:21.510588 15225 raft_consensus.cc:3037] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:21.512943 15225 raft_consensus.cc:504] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6001c4a29f36466390d35a1e3133f756" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 44523 } } peers { permanent_uuid: "79382a0ed9294a32aa664364947c8a06" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 35243 } } peers { permanent_uuid: "e41bbe7b55d64a498c169a52f8abaf1a" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 35945 } }
I20260929 20:21:21.513705 15225 leader_election.cc:288] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a [CANDIDATE]: Term 1 election: Requested vote from peers 6001c4a29f36466390d35a1e3133f756 (127.27.185.65:44523), 79382a0ed9294a32aa664364947c8a06 (127.27.185.66:35243)
I20260929 20:21:21.514217 14893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bca90a3e2a64e8189570793805613e6" candidate_uuid: "e41bbe7b55d64a498c169a52f8abaf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6001c4a29f36466390d35a1e3133f756"
I20260929 20:21:21.514218 15024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bca90a3e2a64e8189570793805613e6" candidate_uuid: "e41bbe7b55d64a498c169a52f8abaf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79382a0ed9294a32aa664364947c8a06"
I20260929 20:21:21.514438 14893 raft_consensus.cc:3037] T 7bca90a3e2a64e8189570793805613e6 P 6001c4a29f36466390d35a1e3133f756 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:21.514469 15024 raft_consensus.cc:3037] T 7bca90a3e2a64e8189570793805613e6 P 79382a0ed9294a32aa664364947c8a06 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:21.517853 15024 raft_consensus.cc:2454] T 7bca90a3e2a64e8189570793805613e6 P 79382a0ed9294a32aa664364947c8a06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e41bbe7b55d64a498c169a52f8abaf1a in term 1.
I20260929 20:21:21.517853 14893 raft_consensus.cc:2454] T 7bca90a3e2a64e8189570793805613e6 P 6001c4a29f36466390d35a1e3133f756 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e41bbe7b55d64a498c169a52f8abaf1a in term 1.
I20260929 20:21:21.518313 15090 leader_election.cc:302] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79382a0ed9294a32aa664364947c8a06, e41bbe7b55d64a498c169a52f8abaf1a; no voters:
I20260929 20:21:21.518698 15225 raft_consensus.cc:2781] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:21:21.519176 15225 raft_consensus.cc:686] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a [term 1 LEADER]: Becoming Leader. State: Replica: e41bbe7b55d64a498c169a52f8abaf1a, State: Running, Role: LEADER
I20260929 20:21:21.519989 15225 consensus_queue.cc:237] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6001c4a29f36466390d35a1e3133f756" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 44523 } } peers { permanent_uuid: "79382a0ed9294a32aa664364947c8a06" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 35243 } } peers { permanent_uuid: "e41bbe7b55d64a498c169a52f8abaf1a" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 35945 } }
I20260929 20:21:21.524559 14752 catalog_manager.cc:5495] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a reported cstate change: term changed from 0 to 1, leader changed from <none> to e41bbe7b55d64a498c169a52f8abaf1a (127.27.185.67). New cstate: current_term: 1 leader_uuid: "e41bbe7b55d64a498c169a52f8abaf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6001c4a29f36466390d35a1e3133f756" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 44523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79382a0ed9294a32aa664364947c8a06" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 35243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e41bbe7b55d64a498c169a52f8abaf1a" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 35945 } health_report { overall_health: HEALTHY } } }
W20260929 20:21:21.525321 15200 tablet.cc:2369] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:21:21.540691 15216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:21.535305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43122 (local address 127.27.185.67:35945)
0929 20:21:21.535502 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:21.535506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:21.535522 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:21:21.535611 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:21.535614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:21.535698 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:21.535827 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:21:21.536163 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.536872 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.538253 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.538897 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.539011 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:21.539626 (+ 615us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:21:21.540177 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:21:21.540312 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:21.540454 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 20:21:21.540491 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 20:21:21.548946 15024 raft_consensus.cc:1261] T 7bca90a3e2a64e8189570793805613e6 P 79382a0ed9294a32aa664364947c8a06 [term 1 FOLLOWER]: Refusing update from remote peer e41bbe7b55d64a498c169a52f8abaf1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:21:21.548978 14893 raft_consensus.cc:1261] T 7bca90a3e2a64e8189570793805613e6 P 6001c4a29f36466390d35a1e3133f756 [term 1 FOLLOWER]: Refusing update from remote peer e41bbe7b55d64a498c169a52f8abaf1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:21:21.549989 15225 consensus_queue.cc:1035] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "79382a0ed9294a32aa664364947c8a06" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 35243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:21:21.550324 15227 consensus_queue.cc:1035] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6001c4a29f36466390d35a1e3133f756" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 44523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 20:21:21.560330 15070 tablet.cc:2369] T 7bca90a3e2a64e8189570793805613e6 P 79382a0ed9294a32aa664364947c8a06: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:21:21.569166 15232 mvcc.cc:205] Tried to move back new op lower bound from 7334761601214009344 to 7334761601109762048. Current Snapshot: MvccSnapshot[applied={T|T < 7334761601214009344}]
I20260929 20:21:21.576838 14751 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:21:21.577152 14751 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:21:21.581058 14751 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7d39f74b12904d6f9eaaf93a75541b40: Sending DeleteTablet for 3 replicas of tablet 7bca90a3e2a64e8189570793805613e6
I20260929 20:21:21.582356 15134 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bca90a3e2a64e8189570793805613e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:21:21 UTC) from {username='slave'} at 127.0.0.1:43118
I20260929 20:21:21.582455 15004 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bca90a3e2a64e8189570793805613e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:21:21 UTC) from {username='slave'} at 127.0.0.1:44236
I20260929 20:21:21.582585 14873 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bca90a3e2a64e8189570793805613e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:21:21 UTC) from {username='slave'} at 127.0.0.1:54420
I20260929 20:21:21.584312 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 14811
I20260929 20:21:21.594360 15239 tablet_replica.cc:330] T 7bca90a3e2a64e8189570793805613e6 P 79382a0ed9294a32aa664364947c8a06: stopping tablet replica
I20260929 20:21:21.595023 15239 raft_consensus.cc:2229] T 7bca90a3e2a64e8189570793805613e6 P 79382a0ed9294a32aa664364947c8a06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:21:21.595664 15239 raft_consensus.cc:2258] T 7bca90a3e2a64e8189570793805613e6 P 79382a0ed9294a32aa664364947c8a06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:21:21.596590 15240 tablet_replica.cc:330] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a: stopping tablet replica
I20260929 20:21:21.597126 15241 tablet_replica.cc:330] T 7bca90a3e2a64e8189570793805613e6 P 6001c4a29f36466390d35a1e3133f756: stopping tablet replica
I20260929 20:21:21.597091 15240 raft_consensus.cc:2229] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:21:21.597580 15241 raft_consensus.cc:2229] T 7bca90a3e2a64e8189570793805613e6 P 6001c4a29f36466390d35a1e3133f756 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:21:21.597803 15240 raft_consensus.cc:2258] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:21:21.598639 15241 raft_consensus.cc:2258] T 7bca90a3e2a64e8189570793805613e6 P 6001c4a29f36466390d35a1e3133f756 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:21:21.600082 15239 ts_tablet_manager.cc:1905] T 7bca90a3e2a64e8189570793805613e6 P 79382a0ed9294a32aa664364947c8a06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:21:21.602452 15241 ts_tablet_manager.cc:1905] T 7bca90a3e2a64e8189570793805613e6 P 6001c4a29f36466390d35a1e3133f756: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:21:21.605334 15239 ts_tablet_manager.cc:1918] T 7bca90a3e2a64e8189570793805613e6 P 79382a0ed9294a32aa664364947c8a06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:21:21.605522 15239 log.cc:1196] T 7bca90a3e2a64e8189570793805613e6 P 79382a0ed9294a32aa664364947c8a06: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-1/wal/wals/7bca90a3e2a64e8189570793805613e6
I20260929 20:21:21.606114 15239 ts_tablet_manager.cc:1939] T 7bca90a3e2a64e8189570793805613e6 P 79382a0ed9294a32aa664364947c8a06: Deleting consensus metadata
I20260929 20:21:21.607262 15240 ts_tablet_manager.cc:1905] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:21:21.608184 14739 catalog_manager.cc:4841] TS 79382a0ed9294a32aa664364947c8a06 (127.27.185.66:35243): tablet 7bca90a3e2a64e8189570793805613e6 (table test-table [id=7505efe4851b45fb883e2e5a84621efd]) successfully deleted
I20260929 20:21:21.610299 15211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:21.585584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54424 (local address 127.27.185.65:44523)
0929 20:21:21.585768 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:21.585775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:21.592687 (+ 6912us) server_negotiation.cc:407] Connection header received
0929 20:21:21.593705 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:21.593711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:21.593841 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:21.593923 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:21.595163 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.596474 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.599355 (+ 2881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.600069 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.600184 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:21.608016 (+ 7832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:21.608031 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:21.608051 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:21.608875 (+ 824us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:21.608952 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:21.608983 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:21.609129 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:21.609770 (+ 641us) server_negotiation.cc:299] Negotiation successful
0929 20:21:21.609905 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:21:21.611160 14933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:21.615607 15240 ts_tablet_manager.cc:1918] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:21:21.615804 15240 log.cc:1196] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-2/wal/wals/7bca90a3e2a64e8189570793805613e6
I20260929 20:21:21.616334 15240 ts_tablet_manager.cc:1939] T 7bca90a3e2a64e8189570793805613e6 P e41bbe7b55d64a498c169a52f8abaf1a: Deleting consensus metadata
I20260929 20:21:21.617913 14737 catalog_manager.cc:4841] TS e41bbe7b55d64a498c169a52f8abaf1a (127.27.185.67:35945): tablet 7bca90a3e2a64e8189570793805613e6 (table test-table [id=7505efe4851b45fb883e2e5a84621efd]) successfully deleted
I20260929 20:21:21.706486 15241 ts_tablet_manager.cc:1918] T 7bca90a3e2a64e8189570793805613e6 P 6001c4a29f36466390d35a1e3133f756: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:21:21.706789 15241 log.cc:1196] T 7bca90a3e2a64e8189570793805613e6 P 6001c4a29f36466390d35a1e3133f756: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790712966917858-28389-0/minicluster-data/ts-0/wal/wals/7bca90a3e2a64e8189570793805613e6
I20260929 20:21:21.707366 15241 ts_tablet_manager.cc:1939] T 7bca90a3e2a64e8189570793805613e6 P 6001c4a29f36466390d35a1e3133f756: Deleting consensus metadata
I20260929 20:21:21.707583 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 14811
I20260929 20:21:21.726254 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 14942
W20260929 20:21:21.726290 14737 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.185.65:44523 (error 108)
I20260929 20:21:21.728554 15212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:21.728026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:44523 (local address 127.0.0.1:54434)
0929 20:21:21.728231 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:21.728363 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 20:21:21.728845 14737 catalog_manager.cc:4584] TS 6001c4a29f36466390d35a1e3133f756 (127.27.185.65:44523): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7bca90a3e2a64e8189570793805613e6: Network error: Client connection negotiation failed: client connection to 127.27.185.65:44523: connect: Connection refused (error 111)
I20260929 20:21:21.734587 15214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:21.727531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44242 (local address 127.27.185.66:35243)
0929 20:21:21.727749 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:21.727754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:21.727869 (+ 115us) server_negotiation.cc:407] Connection header received
0929 20:21:21.728075 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:21.728081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:21.728187 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:21.728319 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:21.728800 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.729657 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.730992 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.731722 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.731859 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:21.732880 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:21.732893 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:21.732913 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:21.733750 (+ 837us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:21.733815 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:21.733840 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:21.733963 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:21.734089 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 20:21:21.734215 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:21:21.735311 15064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:21.782709 15212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:21.782148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:44523 (local address 127.0.0.1:54446)
0929 20:21:21.782414 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:21.782511 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260929 20:21:21.783929 15212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:21.783528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:44523 (local address 127.0.0.1:54450)
0929 20:21:21.783721 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:21.783792 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 20:21:21.838971 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 14942
I20260929 20:21:21.852690 15212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:21.852109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:44523 (local address 127.0.0.1:54452)
0929 20:21:21.852359 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:21.852484 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 20:21:21.854100 15212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:21.853604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:44523 (local address 127.0.0.1:54464)
0929 20:21:21.853806 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:21.853911 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 20:21:21.858434 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 15072
I20260929 20:21:21.866768 15216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:21.859217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43132 (local address 127.27.185.67:35945)
0929 20:21:21.859444 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:21.859450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:21.859478 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:21:21.859716 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:21.859720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:21.859816 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:21.859926 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:21.860417 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.861190 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.863315 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:21.863879 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:21.863978 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:21.865285 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:21.865295 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:21.865310 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:21.865934 (+ 624us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:21.865983 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:21.866003 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:21.866097 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:21.866296 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 20:21:21.866389 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 20:21:21.867553 15194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:21.934552 15212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:21.934061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:44523 (local address 127.0.0.1:54476)
0929 20:21:21.934288 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:21.934376 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 20:21:21.935662 15212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:21.935242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:44523 (local address 127.0.0.1:54478)
0929 20:21:21.935484 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:21.935555 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 20:21:21.990612 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 15072
I20260929 20:21:22.010882 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 14721
I20260929 20:21:22.012118 14782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:22.124842 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 14721
I20260929 20:21:22.124945 15212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:22.124446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:44523 (local address 127.0.0.1:54484)
0929 20:21:22.124655 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:22.124772 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
2026-09-29T20:21:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2862 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:21:22 dist-test-slave-4cx3 krb5kdc[15250](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:21:22 dist-test-slave-4cx3 krb5kdc[15250](info): set up 2 sockets
Sep 29 20:21:22 dist-test-slave-4cx3 krb5kdc[15250](info): commencing operation
krb5kdc: starting...
W20260929 20:21:23.254936 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:21:23 dist-test-slave-4cx3 krb5kdc[15250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:21:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:21:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
I20260929 20:21:23.517027 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:33585
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:43405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:33585
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:21:23.882098 15263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:23.882431 15263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:23.882604 15263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:23.890281 15263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:21:23.890425 15263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:23.890542 15263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:23.890642 15263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:21:23.890705 15263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:21:23.901515 15263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:43405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:33585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:33585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:23.902944 15263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:23.904366 15263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:23.916581 15271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:23.916549 15269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:23.916651 15268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:23.917568 15263 server_base.cc:1034] running on GCE node
I20260929 20:21:23.918406 15263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:23.919965 15263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:23.921209 15263 hybrid_clock.cc:648] HybridClock initialized: now 1790713283921126 us; error 65 us; skew 500 ppm
I20260929 20:21:23.921664 15263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:21:23 dist-test-slave-4cx3 krb5kdc[15250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713283, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:21:23.930886 15263 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:21:23.932709 15263 webserver.cc:460] Webserver started at http://127.27.185.126:33431/ using document root <none> and password file <none>
I20260929 20:21:23.933471 15263 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:23.933625 15263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:23.933972 15263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:23.937170 15263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "1b4199a470794922b14683df26898a72"
format_stamp: "Formatted at 2026-09-29 20:21:23 on dist-test-slave-4cx3"
server_key: "1d1f9eb6070006f762b43a33d9aa2a29"
server_key_iv: "04a1117646fa10d4a8fe527d2c408c2d"
server_key_version: "encryptionkey@0"
I20260929 20:21:23.938009 15263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "1b4199a470794922b14683df26898a72"
format_stamp: "Formatted at 2026-09-29 20:21:23 on dist-test-slave-4cx3"
server_key: "1d1f9eb6070006f762b43a33d9aa2a29"
server_key_iv: "04a1117646fa10d4a8fe527d2c408c2d"
server_key_version: "encryptionkey@0"
I20260929 20:21:23.943672 15263 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:21:23.947340 15278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:23.948812 15263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:21:23.949007 15263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "1b4199a470794922b14683df26898a72"
format_stamp: "Formatted at 2026-09-29 20:21:23 on dist-test-slave-4cx3"
server_key: "1d1f9eb6070006f762b43a33d9aa2a29"
server_key_iv: "04a1117646fa10d4a8fe527d2c408c2d"
server_key_version: "encryptionkey@0"
I20260929 20:21:23.949220 15263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:23.975723 15263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:23.979943 15263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:23.980278 15263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:24.000223 15263 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:33585
I20260929 20:21:24.000248 15329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:33585 every 8 connection(s)
I20260929 20:21:24.001711 15263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:21:24.003327 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 15263
I20260929 20:21:24.003546 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:21:24.004024 28389 external_mini_cluster.cc:1428] Setting key 3735b49c2d2a2cdd489e1019f3800003
I20260929 20:21:24.006858 15330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 20:21:24 dist-test-slave-4cx3 krb5kdc[15250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:21:24.022114 15330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b4199a470794922b14683df26898a72: Bootstrap starting.
I20260929 20:21:24.027065 15330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b4199a470794922b14683df26898a72: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:24.028646 15330 log.cc:824] T 00000000000000000000000000000000 P 1b4199a470794922b14683df26898a72: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:24.032605 15330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b4199a470794922b14683df26898a72: No bootstrap required, opened a new log
I20260929 20:21:24.032749 15333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:24.006051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41056 (local address 127.27.185.126:33585)
0929 20:21:24.006999 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:24.007016 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:24.007076 (+ 60us) server_negotiation.cc:407] Connection header received
0929 20:21:24.009799 (+ 2723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:24.009846 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:24.010463 (+ 617us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:24.011035 (+ 572us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:24.012653 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:24.014991 (+ 2338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:24.016505 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:24.017691 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:24.017909 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:24.022644 (+ 4735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:24.022656 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:24.022679 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:24.022740 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:24.027354 (+ 4614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:24.028212 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:24.028221 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:24.028233 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:24.028492 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:24.029033 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:24.029039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:24.029042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:24.029850 (+ 808us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:24.030115 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:24.030701 (+ 586us) server_negotiation.cc:299] Negotiation successful
0929 20:21:24.031205 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":373,"threads_started":1}
I20260929 20:21:24.041831 15330 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b4199a470794922b14683df26898a72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b4199a470794922b14683df26898a72" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33585 } }
I20260929 20:21:24.042261 15330 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b4199a470794922b14683df26898a72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:24.042405 15330 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b4199a470794922b14683df26898a72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4199a470794922b14683df26898a72, State: Initialized, Role: FOLLOWER
I20260929 20:21:24.043107 15330 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b4199a470794922b14683df26898a72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b4199a470794922b14683df26898a72" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33585 } }
I20260929 20:21:24.043457 15330 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b4199a470794922b14683df26898a72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:21:24.043649 15330 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b4199a470794922b14683df26898a72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:21:24.043808 15330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b4199a470794922b14683df26898a72 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:24.046166 15330 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b4199a470794922b14683df26898a72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b4199a470794922b14683df26898a72" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33585 } }
I20260929 20:21:24.046680 15330 leader_election.cc:302] T 00000000000000000000000000000000 P 1b4199a470794922b14683df26898a72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b4199a470794922b14683df26898a72; no voters:
I20260929 20:21:24.047260 15330 leader_election.cc:288] T 00000000000000000000000000000000 P 1b4199a470794922b14683df26898a72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:21:24.047438 15335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b4199a470794922b14683df26898a72 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:21:24.048146 15335 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b4199a470794922b14683df26898a72 [term 1 LEADER]: Becoming Leader. State: Replica: 1b4199a470794922b14683df26898a72, State: Running, Role: LEADER
I20260929 20:21:24.048730 15335 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b4199a470794922b14683df26898a72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b4199a470794922b14683df26898a72" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33585 } }
I20260929 20:21:24.049618 15330 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b4199a470794922b14683df26898a72 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:21:24.051756 15336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b4199a470794922b14683df26898a72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b4199a470794922b14683df26898a72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b4199a470794922b14683df26898a72" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33585 } } }
I20260929 20:21:24.052253 15336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b4199a470794922b14683df26898a72 [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:24.052266 15337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b4199a470794922b14683df26898a72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b4199a470794922b14683df26898a72. Latest consensus state: current_term: 1 leader_uuid: "1b4199a470794922b14683df26898a72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b4199a470794922b14683df26898a72" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33585 } } }
I20260929 20:21:24.052938 15337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b4199a470794922b14683df26898a72 [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:24.056557 15342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:21:24.062450 15342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:21:24.072302 15342 catalog_manager.cc:1347] Generated new cluster ID: 26576b816f7b4ae3b8fe06e4176f8d13
I20260929 20:21:24.072431 15342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:21:24.087448 15342 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:21:24.089478 15342 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:21:24.100394 15342 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b4199a470794922b14683df26898a72: Generated new TSK 0
I20260929 20:21:24.101576 15342 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
Entry for principal kudu/127.27.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab.
I20260929 20:21:24.178833 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:0
--local_ip_for_outbound_sockets=127.27.185.65
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:33585
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:43405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:21:24.531890 15356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:24.532233 15356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:24.532407 15356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:24.540194 15356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:24.540376 15356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:24.540561 15356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:21:24.550844 15356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:43405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.65.keytab
--principal=kudu/127.27.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:33585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:24.552304 15356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:24.553930 15356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:24.567672 15361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:24.567735 15364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:24.567689 15362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:24.568961 15356 server_base.cc:1034] running on GCE node
I20260929 20:21:24.569657 15356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:24.570688 15356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:24.571906 15356 hybrid_clock.cc:648] HybridClock initialized: now 1790713284571849 us; error 38 us; skew 500 ppm
I20260929 20:21:24.572278 15356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:21:24 dist-test-slave-4cx3 krb5kdc[15250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713284, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:21:24.575630 15356 init.cc:377] Logged in from keytab as kudu/127.27.185.65@KRBTEST.COM (short username kudu)
I20260929 20:21:24.577318 15356 webserver.cc:460] Webserver started at http://127.27.185.65:35421/ using document root <none> and password file <none>
I20260929 20:21:24.578100 15356 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:24.578258 15356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:24.578605 15356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:24.581779 15356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-0/data/instance:
uuid: "ee8eb2451cec480b9b53c4f64c916f7f"
format_stamp: "Formatted at 2026-09-29 20:21:24 on dist-test-slave-4cx3"
server_key: "78a999f4bf571af4175b45ae908c32ef"
server_key_iv: "901f15ee53d82cf2717d36f6288a39f9"
server_key_version: "encryptionkey@0"
I20260929 20:21:24.582607 15356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance:
uuid: "ee8eb2451cec480b9b53c4f64c916f7f"
format_stamp: "Formatted at 2026-09-29 20:21:24 on dist-test-slave-4cx3"
server_key: "78a999f4bf571af4175b45ae908c32ef"
server_key_iv: "901f15ee53d82cf2717d36f6288a39f9"
server_key_version: "encryptionkey@0"
I20260929 20:21:24.588310 15356 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:21:24.591991 15371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:24.593422 15356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:21:24.593621 15356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "ee8eb2451cec480b9b53c4f64c916f7f"
format_stamp: "Formatted at 2026-09-29 20:21:24 on dist-test-slave-4cx3"
server_key: "78a999f4bf571af4175b45ae908c32ef"
server_key_iv: "901f15ee53d82cf2717d36f6288a39f9"
server_key_version: "encryptionkey@0"
I20260929 20:21:24.593839 15356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:24.612138 15356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:24.616586 15356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:24.616940 15356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:24.618026 15356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:21:24.619674 15356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:21:24.619787 15356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:24.619968 15356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:21:24.620054 15356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:24.657542 15356 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:46391
I20260929 20:21:24.657608 15483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:46391 every 8 connection(s)
I20260929 20:21:24.659066 15356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:21:24.669174 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 15356
I20260929 20:21:24.669389 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance
I20260929 20:21:24.669837 28389 external_mini_cluster.cc:1428] Setting key 5283b3de957d30de3d716f84baa618c5
Sep 29 20:21:24 dist-test-slave-4cx3 krb5kdc[15250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713284, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.65@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:21:24.681246 15486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:24.661936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:59313 (local address 127.27.185.126:33585)
0929 20:21:24.662447 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:24.662454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:24.663615 (+ 1161us) server_negotiation.cc:407] Connection header received
0929 20:21:24.665421 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:24.665426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:24.665501 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:24.665610 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:24.667117 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:24.668318 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:24.670129 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:24.670812 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:24.670979 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:24.674810 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:24.674816 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:24.674819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:24.674855 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:24.677544 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:24.678498 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:24.678507 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:24.678512 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:24.678673 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:24.679187 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:24.679195 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:24.679198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:24.679602 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:24.679776 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:24.680578 (+ 802us) server_negotiation.cc:299] Negotiation successful
0929 20:21:24.680844 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":221,"threads_started":1}
I20260929 20:21:24.682677 15487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:24.662642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33585 (local address 127.27.185.65:59313)
0929 20:21:24.663379 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:24.663499 (+ 120us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:24.665154 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:24.665816 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:24.665824 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:24.666344 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:21:24.666932 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:24.666949 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:24.668498 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:24.668502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:24.669968 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:24.669989 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:24.671075 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:24.671084 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:24.671354 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:24.672096 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:21:24.672120 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:21:24.674578 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:21:24.677705 (+ 3127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:21:24.677715 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:21:24.677733 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:21:24.678194 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:21:24.678805 (+ 611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:21:24.678811 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:21:24.678815 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:21:24.679060 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:21:24.679746 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:21:24.679755 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:21:24.680160 (+ 405us) client_negotiation.cc:769] Sending connection context
0929 20:21:24.680477 (+ 317us) client_negotiation.cc:240] Negotiation successful
0929 20:21:24.681130 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":297,"threads_started":1}
I20260929 20:21:24.684722 15484 heartbeater.cc:347] Connected to a master server at 127.27.185.126:33585
I20260929 20:21:24.685048 15484 heartbeater.cc:464] Registering TS with master...
I20260929 20:21:24.685983 15484 heartbeater.cc:511] Master 127.27.185.126:33585 requested a full tablet report, sending...
I20260929 20:21:24.688668 15295 ts_manager.cc:194] Registered new tserver with Master: ee8eb2451cec480b9b53c4f64c916f7f (127.27.185.65:46391)
I20260929 20:21:24.690925 15295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.65@KRBTEST.COM'} at 127.27.185.65:59313
WARNING: no policy specified for kudu/127.27.185.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
Entry for principal kudu/127.27.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab.
I20260929 20:21:24.735162 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:0
--local_ip_for_outbound_sockets=127.27.185.66
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:33585
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:43405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:21:25.075668 15490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:25.075978 15490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:25.076073 15490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:25.083467 15490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:25.083591 15490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:25.083709 15490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:21:25.093767 15490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:43405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.66.keytab
--principal=kudu/127.27.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:33585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:25.095067 15490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:25.096511 15490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:25.110009 15496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:25.110015 15498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:25.110019 15495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:25.110875 15490 server_base.cc:1034] running on GCE node
I20260929 20:21:25.111601 15490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:25.112608 15490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:25.113842 15490 hybrid_clock.cc:648] HybridClock initialized: now 1790713285113793 us; error 45 us; skew 500 ppm
I20260929 20:21:25.114181 15490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:21:25 dist-test-slave-4cx3 krb5kdc[15250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713285, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:21:25.117458 15490 init.cc:377] Logged in from keytab as kudu/127.27.185.66@KRBTEST.COM (short username kudu)
I20260929 20:21:25.118902 15490 webserver.cc:460] Webserver started at http://127.27.185.66:44237/ using document root <none> and password file <none>
I20260929 20:21:25.119652 15490 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:25.119757 15490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:25.120023 15490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:25.122963 15490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-1/data/instance:
uuid: "8c2d9030c76249569efc9ddbf8501588"
format_stamp: "Formatted at 2026-09-29 20:21:25 on dist-test-slave-4cx3"
server_key: "1428fb039c4fcf7e37d7ef922e980eea"
server_key_iv: "4d7c3f798248caf933eb965944c1625d"
server_key_version: "encryptionkey@0"
I20260929 20:21:25.123742 15490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance:
uuid: "8c2d9030c76249569efc9ddbf8501588"
format_stamp: "Formatted at 2026-09-29 20:21:25 on dist-test-slave-4cx3"
server_key: "1428fb039c4fcf7e37d7ef922e980eea"
server_key_iv: "4d7c3f798248caf933eb965944c1625d"
server_key_version: "encryptionkey@0"
I20260929 20:21:25.128983 15490 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:21:25.132781 15505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:25.134238 15490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:21:25.134410 15490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "8c2d9030c76249569efc9ddbf8501588"
format_stamp: "Formatted at 2026-09-29 20:21:25 on dist-test-slave-4cx3"
server_key: "1428fb039c4fcf7e37d7ef922e980eea"
server_key_iv: "4d7c3f798248caf933eb965944c1625d"
server_key_version: "encryptionkey@0"
I20260929 20:21:25.134620 15490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:25.168778 15490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:25.173238 15490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:25.173543 15490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:25.174547 15490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:21:25.176007 15490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:21:25.176090 15490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:25.176236 15490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:21:25.176290 15490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:25.210039 15490 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:37017
I20260929 20:21:25.210095 15617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:37017 every 8 connection(s)
I20260929 20:21:25.211465 15490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:21:25.212255 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 15490
I20260929 20:21:25.212441 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance
I20260929 20:21:25.213008 28389 external_mini_cluster.cc:1428] Setting key 3e02d129b665e5541dfdc5b804b224c0
Sep 29 20:21:25 dist-test-slave-4cx3 krb5kdc[15250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713285, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.66@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:21:25.236442 15620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:25.215233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:59871 (local address 127.27.185.126:33585)
0929 20:21:25.215785 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:25.215791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:25.216330 (+ 539us) server_negotiation.cc:407] Connection header received
0929 20:21:25.219269 (+ 2939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:25.219274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:25.219341 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:25.219522 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:25.221594 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:25.222300 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:25.224875 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:25.225302 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:25.225406 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:25.229157 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:25.229166 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:25.229173 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:25.229221 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:25.232899 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:25.233672 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:25.233681 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:25.233687 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:25.233852 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:25.234293 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:25.234301 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:25.234307 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:25.234701 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:25.234883 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:25.235681 (+ 798us) server_negotiation.cc:299] Negotiation successful
0929 20:21:25.235947 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":264,"threads_started":1}
I20260929 20:21:25.237131 15621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:25.215207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33585 (local address 127.27.185.66:59871)
0929 20:21:25.216147 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:25.216200 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:25.218976 (+ 2776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:25.219773 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:25.219788 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:25.220587 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:21:25.221408 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:25.221434 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:25.222460 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:25.222468 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:25.224640 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:25.224657 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:25.225554 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:25.225560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:25.225767 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:25.226459 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:21:25.226486 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:21:25.228895 (+ 2409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:21:25.233071 (+ 4176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:21:25.233079 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:21:25.233093 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:21:25.233525 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:21:25.233960 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:21:25.233964 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:21:25.233966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:21:25.234163 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:21:25.234844 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:21:25.234851 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:21:25.235253 (+ 402us) client_negotiation.cc:769] Sending connection context
0929 20:21:25.235552 (+ 299us) client_negotiation.cc:240] Negotiation successful
0929 20:21:25.235958 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":433,"threads_started":1}
I20260929 20:21:25.238739 15618 heartbeater.cc:347] Connected to a master server at 127.27.185.126:33585
I20260929 20:21:25.239085 15618 heartbeater.cc:464] Registering TS with master...
I20260929 20:21:25.240012 15618 heartbeater.cc:511] Master 127.27.185.126:33585 requested a full tablet report, sending...
I20260929 20:21:25.241509 15295 ts_manager.cc:194] Registered new tserver with Master: 8c2d9030c76249569efc9ddbf8501588 (127.27.185.66:37017)
I20260929 20:21:25.242350 15295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.66@KRBTEST.COM'} at 127.27.185.66:59871
WARNING: no policy specified for kudu/127.27.185.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
Entry for principal kudu/127.27.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab.
I20260929 20:21:25.275939 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:0
--local_ip_for_outbound_sockets=127.27.185.67
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:33585
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:43405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:21:25.614370 15624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:25.614674 15624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:25.614774 15624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:25.622161 15624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:25.622282 15624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:25.622398 15624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:21:25.632313 15624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:43405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.67.keytab
--principal=kudu/127.27.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:33585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:25.633603 15624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:25.634886 15624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:25.648556 15630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:25.648496 15629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:25.648818 15632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:25.649482 15624 server_base.cc:1034] running on GCE node
I20260929 20:21:25.650277 15624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:25.651322 15624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:25.652570 15624 hybrid_clock.cc:648] HybridClock initialized: now 1790713285652517 us; error 54 us; skew 500 ppm
I20260929 20:21:25.652910 15624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:21:25 dist-test-slave-4cx3 krb5kdc[15250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713285, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:21:25.656301 15624 init.cc:377] Logged in from keytab as kudu/127.27.185.67@KRBTEST.COM (short username kudu)
I20260929 20:21:25.658371 15624 webserver.cc:460] Webserver started at http://127.27.185.67:41295/ using document root <none> and password file <none>
I20260929 20:21:25.659282 15624 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:25.659456 15624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:25.659850 15624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:25.664031 15624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-2/data/instance:
uuid: "3fed029042204173805f4388241d62dd"
format_stamp: "Formatted at 2026-09-29 20:21:25 on dist-test-slave-4cx3"
server_key: "974085f2ad5802feb9701c310b98efa9"
server_key_iv: "3fdeebc78641cd560a4b021aed847da6"
server_key_version: "encryptionkey@0"
I20260929 20:21:25.664920 15624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance:
uuid: "3fed029042204173805f4388241d62dd"
format_stamp: "Formatted at 2026-09-29 20:21:25 on dist-test-slave-4cx3"
server_key: "974085f2ad5802feb9701c310b98efa9"
server_key_iv: "3fdeebc78641cd560a4b021aed847da6"
server_key_version: "encryptionkey@0"
I20260929 20:21:25.671864 15624 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 20:21:25.676020 15639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:25.677493 15624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:21:25.677659 15624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "3fed029042204173805f4388241d62dd"
format_stamp: "Formatted at 2026-09-29 20:21:25 on dist-test-slave-4cx3"
server_key: "974085f2ad5802feb9701c310b98efa9"
server_key_iv: "3fdeebc78641cd560a4b021aed847da6"
server_key_version: "encryptionkey@0"
I20260929 20:21:25.677867 15624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:25.695232 15484 heartbeater.cc:503] Master 127.27.185.126:33585 was elected leader, sending a full tablet report...
I20260929 20:21:25.695241 15624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:25.699896 15624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:25.700145 15624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:25.701125 15624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:21:25.702433 15624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:21:25.702513 15624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:25.702615 15624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:21:25.702652 15624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:25.736251 15624 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:43487
I20260929 20:21:25.736308 15751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:43487 every 8 connection(s)
I20260929 20:21:25.737624 15624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:21:25.742003 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 15624
I20260929 20:21:25.742204 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance
I20260929 20:21:25.742666 28389 external_mini_cluster.cc:1428] Setting key bd6aafd8877228d4935a361b21b2c583
Sep 29 20:21:25 dist-test-slave-4cx3 krb5kdc[15250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713285, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.67@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:21:25.758224 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:25.740447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:35009 (local address 127.27.185.126:33585)
0929 20:21:25.740876 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:25.740880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:25.741910 (+ 1030us) server_negotiation.cc:407] Connection header received
0929 20:21:25.744060 (+ 2150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:25.744065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:25.744170 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:25.744285 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:25.746085 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:25.746844 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:25.748533 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:25.748929 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:25.749076 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:25.752567 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:25.752573 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:25.752576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:25.752609 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:25.755005 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:25.755783 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:25.755788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:25.755790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:25.755901 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:25.756334 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:25.756338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:25.756341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:25.756703 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:25.756859 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:25.757602 (+ 743us) server_negotiation.cc:299] Negotiation successful
0929 20:21:25.757821 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":213,"threads_started":1}
I20260929 20:21:25.759213 15755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:25.741049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33585 (local address 127.27.185.67:35009)
0929 20:21:25.741747 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:25.741790 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:25.743778 (+ 1988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:25.744563 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:25.744575 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:25.745319 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:21:25.745923 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:25.745940 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:25.747004 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:25.747010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:25.748371 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:25.748383 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:25.749162 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:25.749169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:25.749452 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:25.750089 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:21:25.750111 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:21:25.752347 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:21:25.755145 (+ 2798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:21:25.755153 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:21:25.755170 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:21:25.755655 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:21:25.756024 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:21:25.756029 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:21:25.756032 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:21:25.756213 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:21:25.756823 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:21:25.756831 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:21:25.757217 (+ 386us) client_negotiation.cc:769] Sending connection context
0929 20:21:25.757556 (+ 339us) client_negotiation.cc:240] Negotiation successful
0929 20:21:25.758104 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":257,"threads_started":1}
I20260929 20:21:25.760499 15752 heartbeater.cc:347] Connected to a master server at 127.27.185.126:33585
I20260929 20:21:25.760809 15752 heartbeater.cc:464] Registering TS with master...
I20260929 20:21:25.761543 15752 heartbeater.cc:511] Master 127.27.185.126:33585 requested a full tablet report, sending...
I20260929 20:21:25.762864 15295 ts_manager.cc:194] Registered new tserver with Master: 3fed029042204173805f4388241d62dd (127.27.185.67:43487)
I20260929 20:21:25.763823 15295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.185.67@KRBTEST.COM'} at 127.27.185.67:35009
I20260929 20:21:25.769996 28389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:21:25.786471 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:25.774326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41064 (local address 127.27.185.126:33585)
0929 20:21:25.774505 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:25.774510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:25.774924 (+ 414us) server_negotiation.cc:407] Connection header received
0929 20:21:25.775065 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:25.775068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:25.775139 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:25.775222 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:25.776682 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:25.777408 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:25.778625 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:25.778996 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:25.779104 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:25.781514 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:25.781519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:25.781522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:25.781573 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:25.784187 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:25.784810 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:25.784813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:25.784816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:25.784928 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:25.785237 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:25.785239 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:25.785241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:25.785566 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:25.785744 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:25.786059 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 20:21:25.786239 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 20:21:25.790481 15295 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:21:25.797391 15295 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:21:25.821058 15765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:25.809931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:46391 (local address 127.0.0.1:38240)
0929 20:21:25.811246 (+ 1315us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:25.811311 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:25.811564 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:25.813055 (+ 1491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:25.813063 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:25.813101 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:21:25.813563 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:25.813577 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:25.815946 (+ 2369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:25.815952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:25.818264 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:25.818276 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:25.819918 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:25.819925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:25.820115 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:25.820170 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 20:21:25.820296 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 20:21:25.820511 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":914,"mutex_wait_us":114,"thread_start_us":1572,"threads_started":1}
I20260929 20:21:25.821879 15770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:25.811546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43516 (local address 127.27.185.67:43487)
0929 20:21:25.813428 (+ 1882us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:25.813439 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:25.814184 (+ 745us) server_negotiation.cc:407] Connection header received
0929 20:21:25.814269 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:25.814273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:25.814406 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:25.814592 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:21:25.815176 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:25.816452 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:25.819153 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:25.820335 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:25.820443 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:25.820661 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:25.821492 (+ 831us) server_negotiation.cc:299] Negotiation successful
0929 20:21:25.821634 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1692,"thread_start_us":1543,"threads_started":1}
I20260929 20:21:25.821887 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:25.810525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38240 (local address 127.27.185.65:46391)
0929 20:21:25.812330 (+ 1805us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:25.812341 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:25.812374 (+ 33us) server_negotiation.cc:407] Connection header received
0929 20:21:25.812482 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:25.812492 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:25.812772 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:25.813828 (+ 1056us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:21:25.814095 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:25.815807 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:25.818435 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:25.819720 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:25.819863 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:25.821274 (+ 1411us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:25.821358 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 20:21:25.821555 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1617,"thread_start_us":1464,"threads_started":1}
I20260929 20:21:25.821956 15766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:25.810311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:37017 (local address 127.0.0.1:47158)
0929 20:21:25.813063 (+ 2752us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:25.813081 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:25.813205 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:25.814096 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:25.814100 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:25.814125 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:21:25.814519 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:25.814532 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:25.816396 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:25.816400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:25.818908 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:25.818924 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:25.821082 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:25.821089 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:25.821297 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:25.821331 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 20:21:25.821419 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 20:21:25.821548 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2627,"mutex_wait_us":94,"thread_start_us":2464,"threads_started":1}
I20260929 20:21:25.822633 15769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:25.810849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47158 (local address 127.27.185.66:37017)
0929 20:21:25.812635 (+ 1786us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:25.812643 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:25.813346 (+ 703us) server_negotiation.cc:407] Connection header received
0929 20:21:25.813472 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:25.813481 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:25.813702 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:25.813868 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:21:25.814727 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:25.816229 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:25.819085 (+ 2856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:25.820571 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:25.820706 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:25.820894 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:25.822165 (+ 1271us) server_negotiation.cc:299] Negotiation successful
0929 20:21:25.822336 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1650,"thread_start_us":1513,"threads_started":1}
I20260929 20:21:25.822652 15768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:25.810855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:43487 (local address 127.0.0.1:43516)
0929 20:21:25.813908 (+ 3053us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:25.813928 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:25.814048 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:25.814586 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:25.814591 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:25.814614 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:21:25.814965 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:25.814974 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:25.816668 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:25.816676 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:25.818982 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:25.818996 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:25.820805 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:25.820811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:25.821003 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:25.821061 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 20:21:25.822283 (+ 1222us) client_negotiation.cc:240] Negotiation successful
0929 20:21:25.822406 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2947,"mutex_wait_us":4,"thread_start_us":2794,"threads_started":1}
I20260929 20:21:25.826001 15419 tablet_service.cc:1467] Processing CreateTablet for tablet b200e59c7bd142d3a9caec9537b3cff4 (DEFAULT_TABLE table=test-table [id=ec14ecc4ad67488f9b6c0dd7c96afd62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:21:25.826140 15553 tablet_service.cc:1467] Processing CreateTablet for tablet b200e59c7bd142d3a9caec9537b3cff4 (DEFAULT_TABLE table=test-table [id=ec14ecc4ad67488f9b6c0dd7c96afd62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:21:25.826140 15687 tablet_service.cc:1467] Processing CreateTablet for tablet b200e59c7bd142d3a9caec9537b3cff4 (DEFAULT_TABLE table=test-table [id=ec14ecc4ad67488f9b6c0dd7c96afd62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:21:25.827337 15419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b200e59c7bd142d3a9caec9537b3cff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:25.827528 15687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b200e59c7bd142d3a9caec9537b3cff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:25.827539 15553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b200e59c7bd142d3a9caec9537b3cff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:25.839578 15771 tablet_bootstrap.cc:492] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f: Bootstrap starting.
I20260929 20:21:25.843120 15771 tablet_bootstrap.cc:654] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:25.844597 15771 log.cc:824] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:25.844673 15772 tablet_bootstrap.cc:492] T b200e59c7bd142d3a9caec9537b3cff4 P 3fed029042204173805f4388241d62dd: Bootstrap starting.
I20260929 20:21:25.846932 15773 tablet_bootstrap.cc:492] T b200e59c7bd142d3a9caec9537b3cff4 P 8c2d9030c76249569efc9ddbf8501588: Bootstrap starting.
I20260929 20:21:25.848486 15772 tablet_bootstrap.cc:654] T b200e59c7bd142d3a9caec9537b3cff4 P 3fed029042204173805f4388241d62dd: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:25.848930 15771 tablet_bootstrap.cc:492] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f: No bootstrap required, opened a new log
I20260929 20:21:25.849201 15771 ts_tablet_manager.cc:1397] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 20:21:25.849785 15772 log.cc:824] T b200e59c7bd142d3a9caec9537b3cff4 P 3fed029042204173805f4388241d62dd: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:25.850857 15773 tablet_bootstrap.cc:654] T b200e59c7bd142d3a9caec9537b3cff4 P 8c2d9030c76249569efc9ddbf8501588: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:25.852172 15773 log.cc:824] T b200e59c7bd142d3a9caec9537b3cff4 P 8c2d9030c76249569efc9ddbf8501588: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:25.856204 15772 tablet_bootstrap.cc:492] T b200e59c7bd142d3a9caec9537b3cff4 P 3fed029042204173805f4388241d62dd: No bootstrap required, opened a new log
I20260929 20:21:25.856223 15773 tablet_bootstrap.cc:492] T b200e59c7bd142d3a9caec9537b3cff4 P 8c2d9030c76249569efc9ddbf8501588: No bootstrap required, opened a new log
I20260929 20:21:25.856505 15772 ts_tablet_manager.cc:1397] T b200e59c7bd142d3a9caec9537b3cff4 P 3fed029042204173805f4388241d62dd: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20260929 20:21:25.856513 15773 ts_tablet_manager.cc:1397] T b200e59c7bd142d3a9caec9537b3cff4 P 8c2d9030c76249569efc9ddbf8501588: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 20:21:25.861003 15771 raft_consensus.cc:348] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8eb2451cec480b9b53c4f64c916f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 46391 } } peers { permanent_uuid: "8c2d9030c76249569efc9ddbf8501588" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 37017 } } peers { permanent_uuid: "3fed029042204173805f4388241d62dd" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 43487 } }
I20260929 20:21:25.861492 15771 raft_consensus.cc:374] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:25.861596 15771 raft_consensus.cc:729] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee8eb2451cec480b9b53c4f64c916f7f, State: Initialized, Role: FOLLOWER
I20260929 20:21:25.862293 15771 consensus_queue.cc:260] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8eb2451cec480b9b53c4f64c916f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 46391 } } peers { permanent_uuid: "8c2d9030c76249569efc9ddbf8501588" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 37017 } } peers { permanent_uuid: "3fed029042204173805f4388241d62dd" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 43487 } }
I20260929 20:21:25.865789 15771 ts_tablet_manager.cc:1428] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f: Time spent starting tablet: real 0.016s user 0.013s sys 0.003s
I20260929 20:21:25.866223 15772 raft_consensus.cc:348] T b200e59c7bd142d3a9caec9537b3cff4 P 3fed029042204173805f4388241d62dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8eb2451cec480b9b53c4f64c916f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 46391 } } peers { permanent_uuid: "8c2d9030c76249569efc9ddbf8501588" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 37017 } } peers { permanent_uuid: "3fed029042204173805f4388241d62dd" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 43487 } }
I20260929 20:21:25.866732 15772 raft_consensus.cc:374] T b200e59c7bd142d3a9caec9537b3cff4 P 3fed029042204173805f4388241d62dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:25.866869 15772 raft_consensus.cc:729] T b200e59c7bd142d3a9caec9537b3cff4 P 3fed029042204173805f4388241d62dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fed029042204173805f4388241d62dd, State: Initialized, Role: FOLLOWER
I20260929 20:21:25.867621 15772 consensus_queue.cc:260] T b200e59c7bd142d3a9caec9537b3cff4 P 3fed029042204173805f4388241d62dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8eb2451cec480b9b53c4f64c916f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 46391 } } peers { permanent_uuid: "8c2d9030c76249569efc9ddbf8501588" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 37017 } } peers { permanent_uuid: "3fed029042204173805f4388241d62dd" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 43487 } }
I20260929 20:21:25.867599 15773 raft_consensus.cc:348] T b200e59c7bd142d3a9caec9537b3cff4 P 8c2d9030c76249569efc9ddbf8501588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8eb2451cec480b9b53c4f64c916f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 46391 } } peers { permanent_uuid: "8c2d9030c76249569efc9ddbf8501588" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 37017 } } peers { permanent_uuid: "3fed029042204173805f4388241d62dd" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 43487 } }
I20260929 20:21:25.868023 15773 raft_consensus.cc:374] T b200e59c7bd142d3a9caec9537b3cff4 P 8c2d9030c76249569efc9ddbf8501588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:25.868155 15773 raft_consensus.cc:729] T b200e59c7bd142d3a9caec9537b3cff4 P 8c2d9030c76249569efc9ddbf8501588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c2d9030c76249569efc9ddbf8501588, State: Initialized, Role: FOLLOWER
I20260929 20:21:25.868970 15752 heartbeater.cc:503] Master 127.27.185.126:33585 was elected leader, sending a full tablet report...
I20260929 20:21:25.868885 15773 consensus_queue.cc:260] T b200e59c7bd142d3a9caec9537b3cff4 P 8c2d9030c76249569efc9ddbf8501588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8eb2451cec480b9b53c4f64c916f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 46391 } } peers { permanent_uuid: "8c2d9030c76249569efc9ddbf8501588" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 37017 } } peers { permanent_uuid: "3fed029042204173805f4388241d62dd" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 43487 } }
I20260929 20:21:25.869748 15772 ts_tablet_manager.cc:1428] T b200e59c7bd142d3a9caec9537b3cff4 P 3fed029042204173805f4388241d62dd: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260929 20:21:25.869997 15618 heartbeater.cc:503] Master 127.27.185.126:33585 was elected leader, sending a full tablet report...
I20260929 20:21:25.872066 15773 ts_tablet_manager.cc:1428] T b200e59c7bd142d3a9caec9537b3cff4 P 8c2d9030c76249569efc9ddbf8501588: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260929 20:21:25.911872 15485 tablet.cc:2369] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:21:25.963740 15619 tablet.cc:2369] T b200e59c7bd142d3a9caec9537b3cff4 P 8c2d9030c76249569efc9ddbf8501588: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:21:25.989054 15777 raft_consensus.cc:482] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 20:21:25.989374 15753 tablet.cc:2369] T b200e59c7bd142d3a9caec9537b3cff4 P 3fed029042204173805f4388241d62dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:21:25.989418 15777 raft_consensus.cc:504] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8eb2451cec480b9b53c4f64c916f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 46391 } } peers { permanent_uuid: "8c2d9030c76249569efc9ddbf8501588" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 37017 } } peers { permanent_uuid: "3fed029042204173805f4388241d62dd" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 43487 } }
I20260929 20:21:25.991240 15777 leader_election.cc:288] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c2d9030c76249569efc9ddbf8501588 (127.27.185.66:37017), 3fed029042204173805f4388241d62dd (127.27.185.67:43487)
I20260929 20:21:26.000640 15770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:25.991715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:39373 (local address 127.27.185.67:43487)
0929 20:21:25.991917 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:25.991924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:25.992675 (+ 751us) server_negotiation.cc:407] Connection header received
0929 20:21:25.992882 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:25.992888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:25.992966 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:25.993075 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:21:25.994066 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:25.995136 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:25.998205 (+ 3069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:25.999571 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:25.999713 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:25.999781 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:26.000275 (+ 494us) server_negotiation.cc:299] Negotiation successful
0929 20:21:26.000397 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:21:26.000630 15769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:25.991797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:55883 (local address 127.27.185.66:37017)
0929 20:21:25.992025 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:25.992033 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:25.992676 (+ 643us) server_negotiation.cc:407] Connection header received
0929 20:21:25.992882 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:25.992888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:25.992966 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:25.993083 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:21:25.993974 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:25.995130 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:25.998219 (+ 3089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:25.999533 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:25.999705 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:25.999781 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:26.000232 (+ 451us) server_negotiation.cc:299] Negotiation successful
0929 20:21:26.000370 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 20:21:26.001075 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:25.991559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:37017 (local address 127.27.185.65:55883)
0929 20:21:25.992524 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:25.992547 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:25.992746 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:25.993141 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:25.993147 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:25.993181 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:21:25.993794 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:25.993813 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:25.995300 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:25.995305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:25.998030 (+ 2725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:25.998046 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:25.999775 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:25.999782 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:26.000028 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:26.000071 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 20:21:26.000176 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 20:21:26.000333 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"mutex_wait_us":165,"spinlock_wait_cycles":22784,"thread_start_us":641,"threads_started":1}
I20260929 20:21:26.001456 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:25.991624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:43487 (local address 127.27.185.65:39373)
0929 20:21:25.992563 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:25.992582 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:25.992721 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:25.993141 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:25.993147 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:25.993212 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:21:25.993900 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:25.993915 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:25.995299 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:25.995305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:25.998030 (+ 2725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:25.998046 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:25.999948 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:25.999953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:26.000080 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:26.000103 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:21:26.000842 (+ 739us) client_negotiation.cc:240] Negotiation successful
0929 20:21:26.000970 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"mutex_wait_us":222,"spinlock_wait_cycles":33536,"thread_start_us":486,"threads_started":1}
I20260929 20:21:26.002308 15573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b200e59c7bd142d3a9caec9537b3cff4" candidate_uuid: "ee8eb2451cec480b9b53c4f64c916f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c2d9030c76249569efc9ddbf8501588" is_pre_election: true
I20260929 20:21:26.002446 15707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b200e59c7bd142d3a9caec9537b3cff4" candidate_uuid: "ee8eb2451cec480b9b53c4f64c916f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fed029042204173805f4388241d62dd" is_pre_election: true
I20260929 20:21:26.002862 15573 raft_consensus.cc:2454] T b200e59c7bd142d3a9caec9537b3cff4 P 8c2d9030c76249569efc9ddbf8501588 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee8eb2451cec480b9b53c4f64c916f7f in term 0.
I20260929 20:21:26.002941 15707 raft_consensus.cc:2454] T b200e59c7bd142d3a9caec9537b3cff4 P 3fed029042204173805f4388241d62dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee8eb2451cec480b9b53c4f64c916f7f in term 0.
I20260929 20:21:26.003635 15375 leader_election.cc:302] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c2d9030c76249569efc9ddbf8501588, ee8eb2451cec480b9b53c4f64c916f7f; no voters:
I20260929 20:21:26.004099 15777 raft_consensus.cc:2781] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:21:26.004244 15777 raft_consensus.cc:482] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:21:26.004336 15777 raft_consensus.cc:3037] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:26.007467 15777 raft_consensus.cc:504] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8eb2451cec480b9b53c4f64c916f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 46391 } } peers { permanent_uuid: "8c2d9030c76249569efc9ddbf8501588" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 37017 } } peers { permanent_uuid: "3fed029042204173805f4388241d62dd" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 43487 } }
I20260929 20:21:26.008188 15777 leader_election.cc:288] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f [CANDIDATE]: Term 1 election: Requested vote from peers 8c2d9030c76249569efc9ddbf8501588 (127.27.185.66:37017), 3fed029042204173805f4388241d62dd (127.27.185.67:43487)
I20260929 20:21:26.008671 15573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b200e59c7bd142d3a9caec9537b3cff4" candidate_uuid: "ee8eb2451cec480b9b53c4f64c916f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c2d9030c76249569efc9ddbf8501588"
I20260929 20:21:26.008709 15707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b200e59c7bd142d3a9caec9537b3cff4" candidate_uuid: "ee8eb2451cec480b9b53c4f64c916f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fed029042204173805f4388241d62dd"
I20260929 20:21:26.008889 15573 raft_consensus.cc:3037] T b200e59c7bd142d3a9caec9537b3cff4 P 8c2d9030c76249569efc9ddbf8501588 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:26.008908 15707 raft_consensus.cc:3037] T b200e59c7bd142d3a9caec9537b3cff4 P 3fed029042204173805f4388241d62dd [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:26.011092 15707 raft_consensus.cc:2454] T b200e59c7bd142d3a9caec9537b3cff4 P 3fed029042204173805f4388241d62dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee8eb2451cec480b9b53c4f64c916f7f in term 1.
I20260929 20:21:26.011569 15372 leader_election.cc:302] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fed029042204173805f4388241d62dd, ee8eb2451cec480b9b53c4f64c916f7f; no voters:
I20260929 20:21:26.011857 15777 raft_consensus.cc:2781] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:21:26.012086 15573 raft_consensus.cc:2454] T b200e59c7bd142d3a9caec9537b3cff4 P 8c2d9030c76249569efc9ddbf8501588 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee8eb2451cec480b9b53c4f64c916f7f in term 1.
I20260929 20:21:26.012490 15777 raft_consensus.cc:686] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f [term 1 LEADER]: Becoming Leader. State: Replica: ee8eb2451cec480b9b53c4f64c916f7f, State: Running, Role: LEADER
I20260929 20:21:26.013011 15777 consensus_queue.cc:237] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8eb2451cec480b9b53c4f64c916f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 46391 } } peers { permanent_uuid: "8c2d9030c76249569efc9ddbf8501588" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 37017 } } peers { permanent_uuid: "3fed029042204173805f4388241d62dd" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 43487 } }
I20260929 20:21:26.017596 15294 catalog_manager.cc:5495] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f reported cstate change: term changed from 0 to 1, leader changed from <none> to ee8eb2451cec480b9b53c4f64c916f7f (127.27.185.65). New cstate: current_term: 1 leader_uuid: "ee8eb2451cec480b9b53c4f64c916f7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8eb2451cec480b9b53c4f64c916f7f" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 46391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c2d9030c76249569efc9ddbf8501588" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 37017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fed029042204173805f4388241d62dd" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 43487 } health_report { overall_health: UNKNOWN } } }
I20260929 20:21:26.063036 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:26.057482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38244 (local address 127.27.185.65:46391)
0929 20:21:26.057697 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:26.057703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:26.057803 (+ 100us) server_negotiation.cc:407] Connection header received
0929 20:21:26.058071 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:26.058075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:26.058155 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:26.058309 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:21:26.058689 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:26.059473 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:26.060851 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:26.061268 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:26.061401 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:26.061913 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:21:26.062448 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:21:26.062616 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:26.062676 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 20:21:26.062761 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:21:26.071599 15573 raft_consensus.cc:1261] T b200e59c7bd142d3a9caec9537b3cff4 P 8c2d9030c76249569efc9ddbf8501588 [term 1 FOLLOWER]: Refusing update from remote peer ee8eb2451cec480b9b53c4f64c916f7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:21:26.071597 15707 raft_consensus.cc:1261] T b200e59c7bd142d3a9caec9537b3cff4 P 3fed029042204173805f4388241d62dd [term 1 FOLLOWER]: Refusing update from remote peer ee8eb2451cec480b9b53c4f64c916f7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:21:26.072594 15782 consensus_queue.cc:1035] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fed029042204173805f4388241d62dd" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 43487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:21:26.072935 15777 consensus_queue.cc:1035] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c2d9030c76249569efc9ddbf8501588" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 37017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:21:26.085386 15786 mvcc.cc:205] Tried to move back new op lower bound from 7334761619739004928 to 7334761619513143296. Current Snapshot: MvccSnapshot[applied={T|T < 7334761619739004928}]
I20260929 20:21:26.101065 15294 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:21:26.101348 15294 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:21:26.105122 15294 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b4199a470794922b14683df26898a72: Sending DeleteTablet for 3 replicas of tablet b200e59c7bd142d3a9caec9537b3cff4
I20260929 20:21:26.106283 15419 tablet_service.cc:1514] Processing DeleteTablet for tablet b200e59c7bd142d3a9caec9537b3cff4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:21:26 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:38240
I20260929 20:21:26.106328 15553 tablet_service.cc:1514] Processing DeleteTablet for tablet b200e59c7bd142d3a9caec9537b3cff4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:21:26 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:47158
I20260929 20:21:26.107031 15687 tablet_service.cc:1514] Processing DeleteTablet for tablet b200e59c7bd142d3a9caec9537b3cff4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:21:26 UTC) from {username='kudu', principal='kudu/127.27.185.126@KRBTEST.COM'} at 127.0.0.1:43516
I20260929 20:21:26.109210 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 15356
I20260929 20:21:26.116014 15796 tablet_replica.cc:330] T b200e59c7bd142d3a9caec9537b3cff4 P 8c2d9030c76249569efc9ddbf8501588: stopping tablet replica
I20260929 20:21:26.116588 15796 raft_consensus.cc:2229] T b200e59c7bd142d3a9caec9537b3cff4 P 8c2d9030c76249569efc9ddbf8501588 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:21:26.117260 15796 raft_consensus.cc:2258] T b200e59c7bd142d3a9caec9537b3cff4 P 8c2d9030c76249569efc9ddbf8501588 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:21:26.119304 15795 tablet_replica.cc:330] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f: stopping tablet replica
I20260929 20:21:26.119745 15795 raft_consensus.cc:2229] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:21:26.120409 15795 raft_consensus.cc:2258] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:21:26.122785 15795 ts_tablet_manager.cc:1905] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:21:26.129355 15796 ts_tablet_manager.cc:1905] T b200e59c7bd142d3a9caec9537b3cff4 P 8c2d9030c76249569efc9ddbf8501588: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:21:26.130415 15795 ts_tablet_manager.cc:1918] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:21:26.130590 15795 log.cc:1196] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-0/wal/wals/b200e59c7bd142d3a9caec9537b3cff4
I20260929 20:21:26.131108 15795 ts_tablet_manager.cc:1939] T b200e59c7bd142d3a9caec9537b3cff4 P ee8eb2451cec480b9b53c4f64c916f7f: Deleting consensus metadata
I20260929 20:21:26.131515 15797 tablet_replica.cc:330] T b200e59c7bd142d3a9caec9537b3cff4 P 3fed029042204173805f4388241d62dd: stopping tablet replica
I20260929 20:21:26.132835 15797 raft_consensus.cc:2229] T b200e59c7bd142d3a9caec9537b3cff4 P 3fed029042204173805f4388241d62dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:21:26.133080 15280 catalog_manager.cc:4841] TS ee8eb2451cec480b9b53c4f64c916f7f (127.27.185.65:46391): tablet b200e59c7bd142d3a9caec9537b3cff4 (table test-table [id=ec14ecc4ad67488f9b6c0dd7c96afd62]) successfully deleted
I20260929 20:21:26.134615 15797 raft_consensus.cc:2258] T b200e59c7bd142d3a9caec9537b3cff4 P 3fed029042204173805f4388241d62dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:21:26.135473 15796 ts_tablet_manager.cc:1918] T b200e59c7bd142d3a9caec9537b3cff4 P 8c2d9030c76249569efc9ddbf8501588: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:21:26.135648 15796 log.cc:1196] T b200e59c7bd142d3a9caec9537b3cff4 P 8c2d9030c76249569efc9ddbf8501588: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-1/wal/wals/b200e59c7bd142d3a9caec9537b3cff4
I20260929 20:21:26.136137 15796 ts_tablet_manager.cc:1939] T b200e59c7bd142d3a9caec9537b3cff4 P 8c2d9030c76249569efc9ddbf8501588: Deleting consensus metadata
I20260929 20:21:26.137684 15282 catalog_manager.cc:4841] TS 8c2d9030c76249569efc9ddbf8501588 (127.27.185.66:37017): tablet b200e59c7bd142d3a9caec9537b3cff4 (table test-table [id=ec14ecc4ad67488f9b6c0dd7c96afd62]) successfully deleted
Sep 29 20:21:26 dist-test-slave-4cx3 krb5kdc[15250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.65@KRBTEST.COM
I20260929 20:21:26.139787 15797 ts_tablet_manager.cc:1905] T b200e59c7bd142d3a9caec9537b3cff4 P 3fed029042204173805f4388241d62dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:21:26.146340 15797 ts_tablet_manager.cc:1918] T b200e59c7bd142d3a9caec9537b3cff4 P 3fed029042204173805f4388241d62dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:21:26.146530 15797 log.cc:1196] T b200e59c7bd142d3a9caec9537b3cff4 P 3fed029042204173805f4388241d62dd: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790712966917858-28389-0/minicluster-data/ts-2/wal/wals/b200e59c7bd142d3a9caec9537b3cff4
I20260929 20:21:26.147092 15797 ts_tablet_manager.cc:1939] T b200e59c7bd142d3a9caec9537b3cff4 P 3fed029042204173805f4388241d62dd: Deleting consensus metadata
I20260929 20:21:26.148074 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:26.110948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38250 (local address 127.27.185.65:46391)
0929 20:21:26.111204 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:26.111214 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:26.123951 (+ 12737us) server_negotiation.cc:407] Connection header received
0929 20:21:26.124531 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:26.124539 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:26.124710 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:26.124813 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:26.127183 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:26.128987 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:26.133353 (+ 4366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:26.134079 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:26.134191 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:26.140411 (+ 6220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:26.140428 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:26.140445 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:26.140507 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:26.144693 (+ 4186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:26.145592 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:26.145601 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:26.145605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:26.145794 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:26.146268 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:26.146272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:26.146275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:26.146777 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:26.147096 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:26.147450 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 20:21:26.147708 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 20:21:26.148933 15479 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:26.149119 15279 catalog_manager.cc:4841] TS 3fed029042204173805f4388241d62dd (127.27.185.67:43487): tablet b200e59c7bd142d3a9caec9537b3cff4 (table test-table [id=ec14ecc4ad67488f9b6c0dd7c96afd62]) successfully deleted
I20260929 20:21:26.249153 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 15356
I20260929 20:21:26.265188 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 15490
Sep 29 20:21:26 dist-test-slave-4cx3 krb5kdc[15250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.66@KRBTEST.COM
I20260929 20:21:26.279990 15769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:26.265952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47160 (local address 127.27.185.66:37017)
0929 20:21:26.266134 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:26.266138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:26.266170 (+ 32us) server_negotiation.cc:407] Connection header received
0929 20:21:26.266342 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:26.266346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:26.266419 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:26.266527 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:26.267898 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:26.268648 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:26.269864 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:26.270229 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:26.270322 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:26.274720 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:26.274736 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:26.274753 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:26.274791 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:26.277557 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:26.278187 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:26.278194 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:26.278197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:26.278314 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:26.278671 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:26.278674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:26.278676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:26.279044 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:26.279267 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:26.279563 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 20:21:26.279726 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:21:26.280664 15613 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:26.372731 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 15490
I20260929 20:21:26.391007 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 15624
Sep 29 20:21:26 dist-test-slave-4cx3 krb5kdc[15250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.67@KRBTEST.COM
I20260929 20:21:26.407316 15770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:26.391945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43518 (local address 127.27.185.67:43487)
0929 20:21:26.392123 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:26.392127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:26.392174 (+ 47us) server_negotiation.cc:407] Connection header received
0929 20:21:26.392294 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:26.392298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:26.392362 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:26.392440 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:26.393984 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:26.394703 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:26.396155 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:26.396541 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:26.396635 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:26.401171 (+ 4536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:26.401188 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:26.401205 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:26.401243 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:26.404472 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:26.405302 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:26.405310 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:26.405312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:26.405465 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:26.405852 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:26.405856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:26.405859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:26.406264 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:26.406510 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:26.406847 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 20:21:26.407031 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:21:26.408046 15747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:26.499953 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 15624
I20260929 20:21:26.519699 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 15263
I20260929 20:21:26.520756 15325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:26.630988 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 15263
2026-09-29T20:21:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4509 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T20:21:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:21:26Z Disabled control of system clock
I20260929 20:21:26.713814 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:41625
--webserver_interface=127.27.185.126
--webserver_port=0
--builtin_ntp_servers=127.27.185.84:41019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:41625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 20:21:27.056315 15806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:27.056710 15806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:27.056797 15806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:27.064101 15806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:21:27.064214 15806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:27.064283 15806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:27.064381 15806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:21:27.064426 15806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:21:27.073976 15806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:41019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:41625
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:41625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:27.075476 15806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:27.077610 15806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:27.090157 15811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:27.090095 15812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:27.090191 15814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:27.091012 15806 server_base.cc:1034] running on GCE node
I20260929 20:21:27.091859 15806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:27.093284 15806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:27.094483 15806 hybrid_clock.cc:648] HybridClock initialized: now 1790713287094425 us; error 48 us; skew 500 ppm
I20260929 20:21:27.094861 15806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:21:27.097330 15806 webserver.cc:460] Webserver started at http://127.27.185.126:44823/ using document root <none> and password file <none>
I20260929 20:21:27.098048 15806 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:27.098166 15806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:27.098506 15806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:27.101497 15806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "f0984714a40442e7b6a308058f741037"
format_stamp: "Formatted at 2026-09-29 20:21:27 on dist-test-slave-4cx3"
server_key: "c44a581d229e1f01831e1d014982e24b"
server_key_iv: "f710a63a5873ae61b148bb691752a211"
server_key_version: "encryptionkey@0"
I20260929 20:21:27.102342 15806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "f0984714a40442e7b6a308058f741037"
format_stamp: "Formatted at 2026-09-29 20:21:27 on dist-test-slave-4cx3"
server_key: "c44a581d229e1f01831e1d014982e24b"
server_key_iv: "f710a63a5873ae61b148bb691752a211"
server_key_version: "encryptionkey@0"
I20260929 20:21:27.107713 15806 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:21:27.111467 15820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:27.113076 15806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:21:27.113329 15806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "f0984714a40442e7b6a308058f741037"
format_stamp: "Formatted at 2026-09-29 20:21:27 on dist-test-slave-4cx3"
server_key: "c44a581d229e1f01831e1d014982e24b"
server_key_iv: "f710a63a5873ae61b148bb691752a211"
server_key_version: "encryptionkey@0"
I20260929 20:21:27.113619 15806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:27.134675 15806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:27.135712 15806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:27.136005 15806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:27.155172 15806 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:41625
I20260929 20:21:27.155200 15871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:41625 every 8 connection(s)
I20260929 20:21:27.156674 15806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:21:27.158668 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 15806
I20260929 20:21:27.158865 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:21:27.159350 28389 external_mini_cluster.cc:1428] Setting key ee60723708b4352ba934372b63a8c861
I20260929 20:21:27.161337 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:27.176021 15875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:27.161502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38362 (local address 127.27.185.126:41625)
0929 20:21:27.162662 (+ 1160us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:27.162679 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:27.162737 (+ 58us) server_negotiation.cc:407] Connection header received
0929 20:21:27.164854 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:27.164886 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:27.165618 (+ 732us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:27.166180 (+ 562us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:27.166766 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:27.168929 (+ 2163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:27.170343 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:27.171430 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:27.171632 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:27.172800 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:27.172811 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:27.172832 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:27.173651 (+ 819us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:27.173763 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:27.173798 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:27.173920 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:27.174149 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 20:21:27.174559 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":359,"threads_started":1}
I20260929 20:21:27.177066 15872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0984714a40442e7b6a308058f741037: Bootstrap starting.
I20260929 20:21:27.180372 15872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0984714a40442e7b6a308058f741037: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:27.181445 15872 log.cc:824] T 00000000000000000000000000000000 P f0984714a40442e7b6a308058f741037: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:27.184756 15872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0984714a40442e7b6a308058f741037: No bootstrap required, opened a new log
I20260929 20:21:27.192677 15872 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0984714a40442e7b6a308058f741037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0984714a40442e7b6a308058f741037" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41625 } }
I20260929 20:21:27.193107 15872 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0984714a40442e7b6a308058f741037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:27.193217 15872 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0984714a40442e7b6a308058f741037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0984714a40442e7b6a308058f741037, State: Initialized, Role: FOLLOWER
I20260929 20:21:27.193833 15872 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0984714a40442e7b6a308058f741037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0984714a40442e7b6a308058f741037" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41625 } }
I20260929 20:21:27.194082 15872 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0984714a40442e7b6a308058f741037 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:21:27.194239 15872 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0984714a40442e7b6a308058f741037 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:21:27.194425 15872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0984714a40442e7b6a308058f741037 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:27.197074 15872 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0984714a40442e7b6a308058f741037 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0984714a40442e7b6a308058f741037" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41625 } }
I20260929 20:21:27.197700 15872 leader_election.cc:302] T 00000000000000000000000000000000 P f0984714a40442e7b6a308058f741037 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0984714a40442e7b6a308058f741037; no voters:
I20260929 20:21:27.198272 15872 leader_election.cc:288] T 00000000000000000000000000000000 P f0984714a40442e7b6a308058f741037 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:21:27.198422 15877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0984714a40442e7b6a308058f741037 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:21:27.199126 15877 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0984714a40442e7b6a308058f741037 [term 1 LEADER]: Becoming Leader. State: Replica: f0984714a40442e7b6a308058f741037, State: Running, Role: LEADER
I20260929 20:21:27.199682 15877 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0984714a40442e7b6a308058f741037 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0984714a40442e7b6a308058f741037" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41625 } }
I20260929 20:21:27.200462 15872 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0984714a40442e7b6a308058f741037 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:21:27.202538 15879 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0984714a40442e7b6a308058f741037 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0984714a40442e7b6a308058f741037. Latest consensus state: current_term: 1 leader_uuid: "f0984714a40442e7b6a308058f741037" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0984714a40442e7b6a308058f741037" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41625 } } }
I20260929 20:21:27.202934 15879 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0984714a40442e7b6a308058f741037 [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:27.205063 15878 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0984714a40442e7b6a308058f741037 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0984714a40442e7b6a308058f741037" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0984714a40442e7b6a308058f741037" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41625 } } }
I20260929 20:21:27.205292 15878 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0984714a40442e7b6a308058f741037 [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:27.205287 15883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:21:27.212033 15883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:21:27.222386 15883 catalog_manager.cc:1347] Generated new cluster ID: 769af3e81bbe40d7b57e2c19e928285b
I20260929 20:21:27.222558 15883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:21:27.232182 15883 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:21:27.233556 15883 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:21:27.243656 15883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0984714a40442e7b6a308058f741037: Generated new TSK 0
I20260929 20:21:27.244611 15883 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:21:27.252059 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.65:0
--local_ip_for_outbound_sockets=127.27.185.65
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:41625
--builtin_ntp_servers=127.27.185.84:41019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 20:21:27.598135 15896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:27.598441 15896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:27.598603 15896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:27.606786 15896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:27.606909 15896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:27.607065 15896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.65
I20260929 20:21:27.616513 15896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:41019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.185.65
--webserver_port=0
--tserver_master_addrs=127.27.185.126:41625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.65
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:27.617928 15896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:27.619890 15896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:27.633255 15904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:27.633338 15902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:27.633769 15901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:27.634016 15896 server_base.cc:1034] running on GCE node
I20260929 20:21:27.634684 15896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:27.635684 15896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:27.636901 15896 hybrid_clock.cc:648] HybridClock initialized: now 1790713287636858 us; error 27 us; skew 500 ppm
I20260929 20:21:27.637236 15896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:21:27.639562 15896 webserver.cc:460] Webserver started at http://127.27.185.65:43443/ using document root <none> and password file <none>
I20260929 20:21:27.640288 15896 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:27.640378 15896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:27.640654 15896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:27.643568 15896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-0/data/instance:
uuid: "7e7f6033ced04c2bb63ef288745892b3"
format_stamp: "Formatted at 2026-09-29 20:21:27 on dist-test-slave-4cx3"
server_key: "72a771af643750bdcc380476664b812d"
server_key_iv: "b497ad174ef3aa3c583a4e0c651da7e5"
server_key_version: "encryptionkey@0"
I20260929 20:21:27.644316 15896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance:
uuid: "7e7f6033ced04c2bb63ef288745892b3"
format_stamp: "Formatted at 2026-09-29 20:21:27 on dist-test-slave-4cx3"
server_key: "72a771af643750bdcc380476664b812d"
server_key_iv: "b497ad174ef3aa3c583a4e0c651da7e5"
server_key_version: "encryptionkey@0"
I20260929 20:21:27.649420 15896 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 20:21:27.652868 15910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:27.654217 15896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:21:27.654407 15896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-0/wal
uuid: "7e7f6033ced04c2bb63ef288745892b3"
format_stamp: "Formatted at 2026-09-29 20:21:27 on dist-test-slave-4cx3"
server_key: "72a771af643750bdcc380476664b812d"
server_key_iv: "b497ad174ef3aa3c583a4e0c651da7e5"
server_key_version: "encryptionkey@0"
I20260929 20:21:27.654639 15896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:27.675633 15896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:27.676753 15896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:27.677089 15896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:27.678117 15896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:21:27.679680 15896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:21:27.679806 15896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:27.679932 15896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:21:27.680019 15896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:27.719300 15896 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.65:37247
I20260929 20:21:27.719358 16022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.65:37247 every 8 connection(s)
I20260929 20:21:27.720741 15896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-0/data/info.pb
I20260929 20:21:27.721552 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 15896
I20260929 20:21:27.721767 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-0/wal/instance
I20260929 20:21:27.722422 28389 external_mini_cluster.cc:1428] Setting key 588d5b854e1d7a97e6122e5c4c61ab07
I20260929 20:21:27.725589 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.66:0
--local_ip_for_outbound_sockets=127.27.185.66
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:41625
--builtin_ntp_servers=127.27.185.84:41019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 20:21:27.736160 16025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:27.724121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:55391 (local address 127.27.185.126:41625)
0929 20:21:27.724588 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:27.724593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:27.726195 (+ 1602us) server_negotiation.cc:407] Connection header received
0929 20:21:27.728090 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:27.728099 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:27.728190 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:27.728318 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:27.729460 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:27.730691 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:27.732607 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:27.733386 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:27.733549 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:27.735163 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:27.735169 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:27.735172 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:27.735193 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:27.735237 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:27.735249 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:27.735352 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:27.735662 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 20:21:27.735779 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":262,"threads_started":1}
I20260929 20:21:27.737334 16026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:27.724910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41625 (local address 127.27.185.65:55391)
0929 20:21:27.726007 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:27.726063 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:27.727807 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:27.728517 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:27.728525 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:27.728622 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:21:27.729254 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:27.729272 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:27.730868 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:27.730872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:27.732421 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:27.732436 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:27.733686 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:27.733694 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:27.733966 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:27.734755 (+ 789us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:21:27.734757 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:21:27.734804 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:21:27.734908 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:21:27.735356 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:21:27.735372 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:21:27.735425 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 20:21:27.735647 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 20:21:27.736136 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":352,"threads_started":1}
I20260929 20:21:27.739114 16023 heartbeater.cc:347] Connected to a master server at 127.27.185.126:41625
I20260929 20:21:27.739494 16023 heartbeater.cc:464] Registering TS with master...
I20260929 20:21:27.740285 16023 heartbeater.cc:511] Master 127.27.185.126:41625 requested a full tablet report, sending...
I20260929 20:21:27.742442 15837 ts_manager.cc:194] Registered new tserver with Master: 7e7f6033ced04c2bb63ef288745892b3 (127.27.185.65:37247)
I20260929 20:21:27.744204 15837 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.185.65:55391
W20260929 20:21:28.072963 16027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:28.073315 16027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:28.073475 16027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:28.081027 16027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:28.081203 16027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:28.081387 16027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.66
I20260929 20:21:28.090926 16027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:41019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.185.66
--webserver_port=0
--tserver_master_addrs=127.27.185.126:41625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.66
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:28.092424 16027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:28.094247 16027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:28.107936 16032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:28.107990 16035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:28.108042 16033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:28.109329 16027 server_base.cc:1034] running on GCE node
I20260929 20:21:28.109949 16027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:28.110888 16027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:28.112099 16027 hybrid_clock.cc:648] HybridClock initialized: now 1790713288112031 us; error 56 us; skew 500 ppm
I20260929 20:21:28.112471 16027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:21:28.114684 16027 webserver.cc:460] Webserver started at http://127.27.185.66:41613/ using document root <none> and password file <none>
I20260929 20:21:28.115347 16027 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:28.115492 16027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:28.115777 16027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:28.118721 16027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-1/data/instance:
uuid: "072d308c8bec485381e16193e3191103"
format_stamp: "Formatted at 2026-09-29 20:21:28 on dist-test-slave-4cx3"
server_key: "63a639fb1ba270eb3c2227082160292e"
server_key_iv: "1224fc28e0ae1b847198ce2d07a418d4"
server_key_version: "encryptionkey@0"
I20260929 20:21:28.119509 16027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance:
uuid: "072d308c8bec485381e16193e3191103"
format_stamp: "Formatted at 2026-09-29 20:21:28 on dist-test-slave-4cx3"
server_key: "63a639fb1ba270eb3c2227082160292e"
server_key_iv: "1224fc28e0ae1b847198ce2d07a418d4"
server_key_version: "encryptionkey@0"
I20260929 20:21:28.124559 16027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 20:21:28.128011 16041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:28.129330 16027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:21:28.129498 16027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-1/wal
uuid: "072d308c8bec485381e16193e3191103"
format_stamp: "Formatted at 2026-09-29 20:21:28 on dist-test-slave-4cx3"
server_key: "63a639fb1ba270eb3c2227082160292e"
server_key_iv: "1224fc28e0ae1b847198ce2d07a418d4"
server_key_version: "encryptionkey@0"
I20260929 20:21:28.129714 16027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:28.155977 16027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:28.156844 16027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:28.157168 16027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:28.158215 16027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:21:28.159854 16027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:21:28.159972 16027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:28.160104 16027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:21:28.160192 16027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:28.195158 16027 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.66:42451
I20260929 20:21:28.195247 16153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.66:42451 every 8 connection(s)
I20260929 20:21:28.196544 16027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-1/data/info.pb
I20260929 20:21:28.204236 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 16027
I20260929 20:21:28.204419 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-1/wal/instance
I20260929 20:21:28.204859 28389 external_mini_cluster.cc:1428] Setting key 498c13d131885ac116080d220b4a0304
I20260929 20:21:28.208070 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.185.67:0
--local_ip_for_outbound_sockets=127.27.185.67
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:41625
--builtin_ntp_servers=127.27.185.84:41019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 20:21:28.209894 16025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:28.199081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.66:48061 (local address 127.27.185.126:41625)
0929 20:21:28.199323 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:28.199330 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:28.200869 (+ 1539us) server_negotiation.cc:407] Connection header received
0929 20:21:28.202649 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:28.202653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:28.202723 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:28.202814 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:28.204045 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.204872 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.206814 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.207274 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.207376 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:28.209145 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:28.209150 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:28.209153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:28.209171 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:28.209204 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:28.209215 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:28.209289 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:28.209581 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 20:21:28.209665 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 20:21:28.211547 16156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:28.199734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41625 (local address 127.27.185.66:48061)
0929 20:21:28.200681 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:28.200724 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:28.202386 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:28.203016 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:28.203025 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:28.203146 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:21:28.203851 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:28.203878 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.205083 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.205087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:28.206680 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:28.206693 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.207547 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.207555 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:28.207916 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:28.208761 (+ 845us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:21:28.208767 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:21:28.208825 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:21:28.208941 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:21:28.209329 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:21:28.209344 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:21:28.209356 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 20:21:28.209549 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 20:21:28.210037 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":262,"threads_started":1}
I20260929 20:21:28.213068 16154 heartbeater.cc:347] Connected to a master server at 127.27.185.126:41625
I20260929 20:21:28.213397 16154 heartbeater.cc:464] Registering TS with master...
I20260929 20:21:28.214337 16154 heartbeater.cc:511] Master 127.27.185.126:41625 requested a full tablet report, sending...
I20260929 20:21:28.216043 15837 ts_manager.cc:194] Registered new tserver with Master: 072d308c8bec485381e16193e3191103 (127.27.185.66:42451)
I20260929 20:21:28.217315 15837 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.185.66:48061
W20260929 20:21:28.547145 16157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:28.547550 16157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:28.547740 16157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:28.555189 16157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:28.555331 16157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:28.555532 16157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.185.67
I20260929 20:21:28.564976 16157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:41019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.185.67
--webserver_port=0
--tserver_master_addrs=127.27.185.126:41625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.185.67
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:28.566360 16157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:28.568289 16157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:28.582415 16162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:28.582461 16163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:28.582494 16165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:28.583510 16157 server_base.cc:1034] running on GCE node
I20260929 20:21:28.584395 16157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:28.585453 16157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:28.586690 16157 hybrid_clock.cc:648] HybridClock initialized: now 1790713288586625 us; error 56 us; skew 500 ppm
I20260929 20:21:28.587044 16157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:21:28.589452 16157 webserver.cc:460] Webserver started at http://127.27.185.67:35133/ using document root <none> and password file <none>
I20260929 20:21:28.590204 16157 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:28.590346 16157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:28.590682 16157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:28.593775 16157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-2/data/instance:
uuid: "c8d87365e5b945d3a3bc01249394003f"
format_stamp: "Formatted at 2026-09-29 20:21:28 on dist-test-slave-4cx3"
server_key: "a228db7e3a05c4a19a3621429f16992e"
server_key_iv: "6c393ec4b99f4003851233607715149c"
server_key_version: "encryptionkey@0"
I20260929 20:21:28.594640 16157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance:
uuid: "c8d87365e5b945d3a3bc01249394003f"
format_stamp: "Formatted at 2026-09-29 20:21:28 on dist-test-slave-4cx3"
server_key: "a228db7e3a05c4a19a3621429f16992e"
server_key_iv: "6c393ec4b99f4003851233607715149c"
server_key_version: "encryptionkey@0"
I20260929 20:21:28.600229 16157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:21:28.604029 16171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:28.605408 16157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:21:28.605615 16157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-2/wal
uuid: "c8d87365e5b945d3a3bc01249394003f"
format_stamp: "Formatted at 2026-09-29 20:21:28 on dist-test-slave-4cx3"
server_key: "a228db7e3a05c4a19a3621429f16992e"
server_key_iv: "6c393ec4b99f4003851233607715149c"
server_key_version: "encryptionkey@0"
I20260929 20:21:28.605846 16157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:28.624706 16157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:28.625589 16157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:28.625910 16157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:28.626914 16157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:21:28.628557 16157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:21:28.628691 16157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:28.628816 16157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:21:28.628904 16157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:28.663962 16157 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.67:38575
I20260929 20:21:28.664065 16283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.67:38575 every 8 connection(s)
I20260929 20:21:28.665369 16157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-2/data/info.pb
I20260929 20:21:28.667915 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 16157
I20260929 20:21:28.668131 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-2/wal/instance
I20260929 20:21:28.668655 28389 external_mini_cluster.cc:1428] Setting key 8802f154102fee8bb01c0b68b53cb304
I20260929 20:21:28.679044 16025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:28.668186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.67:45299 (local address 127.27.185.126:41625)
0929 20:21:28.668404 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:28.668410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:28.670093 (+ 1683us) server_negotiation.cc:407] Connection header received
0929 20:21:28.672012 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:28.672018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:28.672113 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:28.672225 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:28.673326 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.674061 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.675932 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.676690 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.676827 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:28.678181 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:28.678187 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:28.678192 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:28.678221 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:28.678253 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:28.678265 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:28.678362 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:28.678607 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 20:21:28.678712 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:21:28.680248 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:28.668819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:41625 (local address 127.27.185.67:45299)
0929 20:21:28.669916 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:28.669972 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:28.671735 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:28.672472 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:28.672481 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:28.672578 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:21:28.673161 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:28.673178 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.674229 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.674234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:28.675691 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:28.675706 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.676922 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.676929 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:28.677189 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:28.677847 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:21:28.677850 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:21:28.677891 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:21:28.677969 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:21:28.678363 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:21:28.678378 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:21:28.678387 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 20:21:28.678596 (+ 209us) client_negotiation.cc:240] Negotiation successful
0929 20:21:28.679030 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":334,"threads_started":1}
I20260929 20:21:28.681686 16284 heartbeater.cc:347] Connected to a master server at 127.27.185.126:41625
I20260929 20:21:28.682035 16284 heartbeater.cc:464] Registering TS with master...
I20260929 20:21:28.682952 16284 heartbeater.cc:511] Master 127.27.185.126:41625 requested a full tablet report, sending...
I20260929 20:21:28.684479 15837 ts_manager.cc:194] Registered new tserver with Master: c8d87365e5b945d3a3bc01249394003f (127.27.185.67:38575)
I20260929 20:21:28.685285 15837 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.185.67:45299
I20260929 20:21:28.696807 28389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:21:28.709939 16025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:28.703022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38372 (local address 127.27.185.126:41625)
0929 20:21:28.703216 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:28.703221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:28.703780 (+ 559us) server_negotiation.cc:407] Connection header received
0929 20:21:28.703945 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:28.703949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:28.704028 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:28.704124 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:28.704779 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.705693 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.707132 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.707707 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.707810 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:28.708969 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:28.708974 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:28.708977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:28.708994 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:28.709027 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:28.709042 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:28.709124 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:28.709526 (+ 402us) server_negotiation.cc:299] Negotiation successful
0929 20:21:28.709676 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:21:28.714566 15837 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:21:28.722502 15837 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:21:28.747663 16023 heartbeater.cc:503] Master 127.27.185.126:41625 was elected leader, sending a full tablet report...
I20260929 20:21:28.762631 16297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:28.736666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58718 (local address 127.27.185.65:37247)
0929 20:21:28.750421 (+ 13755us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:28.750434 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:28.751313 (+ 879us) server_negotiation.cc:407] Connection header received
0929 20:21:28.751453 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:28.751461 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:28.751732 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:28.752788 (+ 1056us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:21:28.753086 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.755138 (+ 2052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.759107 (+ 3969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.760911 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.761051 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:28.761364 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:28.762066 (+ 702us) server_negotiation.cc:299] Negotiation successful
0929 20:21:28.762250 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13511,"thread_start_us":13346,"threads_started":1}
I20260929 20:21:28.762631 16296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:28.736362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:37247 (local address 127.0.0.1:58718)
0929 20:21:28.750855 (+ 14493us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:28.750895 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:28.751104 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:28.751969 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:28.751980 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:28.752014 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:21:28.752494 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:28.752514 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.755331 (+ 2817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.755340 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:28.758778 (+ 3438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:28.758799 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.761509 (+ 2710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.761516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:28.761733 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:28.761794 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 20:21:28.761930 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 20:21:28.762183 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14231,"thread_start_us":14080,"threads_started":1}
I20260929 20:21:28.767324 16301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:28.737585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55188 (local address 127.27.185.67:38575)
0929 20:21:28.758277 (+ 20692us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:28.758287 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:28.759291 (+ 1004us) server_negotiation.cc:407] Connection header received
0929 20:21:28.759496 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:28.759504 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:28.759737 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:28.759961 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:21:28.760904 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.762279 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.765170 (+ 2891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.766110 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.766283 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:28.766584 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:28.766714 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 20:21:28.766875 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20437,"thread_start_us":20270,"threads_started":1}
I20260929 20:21:28.767338 16299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:28.737010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:38575 (local address 127.0.0.1:55188)
0929 20:21:28.759165 (+ 22155us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:28.759180 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:28.759325 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:28.760276 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:28.760280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:28.760304 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:21:28.760689 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:28.760702 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.762469 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.762474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:28.764975 (+ 2501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:28.764987 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.766329 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.766335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:28.766523 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:28.766568 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 20:21:28.766665 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 20:21:28.766823 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22043,"mutex_wait_us":8,"thread_start_us":20335,"threads_started":1}
I20260929 20:21:28.767985 16300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:28.737170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55932 (local address 127.27.185.66:42451)
0929 20:21:28.758746 (+ 21576us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:28.758757 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:28.758791 (+ 34us) server_negotiation.cc:407] Connection header received
0929 20:21:28.758904 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:28.758927 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:28.759253 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:28.759583 (+ 330us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:21:28.760188 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.761803 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.764628 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.766299 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.766436 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:28.766650 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:28.767462 (+ 812us) server_negotiation.cc:299] Negotiation successful
0929 20:21:28.767650 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21392,"thread_start_us":21253,"threads_started":1}
I20260929 20:21:28.767273 15958 tablet_service.cc:1467] Processing CreateTablet for tablet 1467dfaf424441af8bfd4748a9b80c03 (DEFAULT_TABLE table=test-table [id=977c0311d36f4df780c8e0cc34623939]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:21:28.769063 15958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1467dfaf424441af8bfd4748a9b80c03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:28.769213 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:28.736968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:42451 (local address 127.0.0.1:55932)
0929 20:21:28.757070 (+ 20102us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:28.757093 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:28.757246 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:28.759561 (+ 2315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:28.759565 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:28.759589 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:21:28.760001 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:28.760018 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.761965 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.761971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:28.764304 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:28.764324 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.766797 (+ 2473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.766804 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:28.766992 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:28.767022 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 20:21:28.767110 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 20:21:28.767235 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19917,"mutex_wait_us":39,"thread_start_us":21879,"threads_started":1}
I20260929 20:21:28.772589 16219 tablet_service.cc:1467] Processing CreateTablet for tablet 1467dfaf424441af8bfd4748a9b80c03 (DEFAULT_TABLE table=test-table [id=977c0311d36f4df780c8e0cc34623939]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:21:28.773218 16089 tablet_service.cc:1467] Processing CreateTablet for tablet 1467dfaf424441af8bfd4748a9b80c03 (DEFAULT_TABLE table=test-table [id=977c0311d36f4df780c8e0cc34623939]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:21:28.774345 16219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1467dfaf424441af8bfd4748a9b80c03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:28.774739 16089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1467dfaf424441af8bfd4748a9b80c03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:28.783228 16302 tablet_bootstrap.cc:492] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3: Bootstrap starting.
I20260929 20:21:28.786569 16302 tablet_bootstrap.cc:654] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:28.787801 16302 log.cc:824] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:28.790254 16304 tablet_bootstrap.cc:492] T 1467dfaf424441af8bfd4748a9b80c03 P c8d87365e5b945d3a3bc01249394003f: Bootstrap starting.
I20260929 20:21:28.795512 16304 tablet_bootstrap.cc:654] T 1467dfaf424441af8bfd4748a9b80c03 P c8d87365e5b945d3a3bc01249394003f: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:28.796911 16305 tablet_bootstrap.cc:492] T 1467dfaf424441af8bfd4748a9b80c03 P 072d308c8bec485381e16193e3191103: Bootstrap starting.
I20260929 20:21:28.797039 16304 log.cc:824] T 1467dfaf424441af8bfd4748a9b80c03 P c8d87365e5b945d3a3bc01249394003f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:28.799053 16302 tablet_bootstrap.cc:492] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3: No bootstrap required, opened a new log
I20260929 20:21:28.799455 16302 ts_tablet_manager.cc:1397] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.005s
I20260929 20:21:28.800773 16304 tablet_bootstrap.cc:492] T 1467dfaf424441af8bfd4748a9b80c03 P c8d87365e5b945d3a3bc01249394003f: No bootstrap required, opened a new log
I20260929 20:21:28.800806 16305 tablet_bootstrap.cc:654] T 1467dfaf424441af8bfd4748a9b80c03 P 072d308c8bec485381e16193e3191103: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:28.801379 16304 ts_tablet_manager.cc:1397] T 1467dfaf424441af8bfd4748a9b80c03 P c8d87365e5b945d3a3bc01249394003f: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260929 20:21:28.802148 16305 log.cc:824] T 1467dfaf424441af8bfd4748a9b80c03 P 072d308c8bec485381e16193e3191103: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:28.805300 16305 tablet_bootstrap.cc:492] T 1467dfaf424441af8bfd4748a9b80c03 P 072d308c8bec485381e16193e3191103: No bootstrap required, opened a new log
I20260929 20:21:28.805572 16305 ts_tablet_manager.cc:1397] T 1467dfaf424441af8bfd4748a9b80c03 P 072d308c8bec485381e16193e3191103: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 20:21:28.807935 16302 raft_consensus.cc:348] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7f6033ced04c2bb63ef288745892b3" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 37247 } } peers { permanent_uuid: "072d308c8bec485381e16193e3191103" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 42451 } } peers { permanent_uuid: "c8d87365e5b945d3a3bc01249394003f" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 38575 } }
I20260929 20:21:28.808329 16302 raft_consensus.cc:374] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:28.808427 16302 raft_consensus.cc:729] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7f6033ced04c2bb63ef288745892b3, State: Initialized, Role: FOLLOWER
I20260929 20:21:28.809114 16302 consensus_queue.cc:260] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7f6033ced04c2bb63ef288745892b3" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 37247 } } peers { permanent_uuid: "072d308c8bec485381e16193e3191103" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 42451 } } peers { permanent_uuid: "c8d87365e5b945d3a3bc01249394003f" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 38575 } }
I20260929 20:21:28.811193 16302 ts_tablet_manager.cc:1428] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260929 20:21:28.812101 16304 raft_consensus.cc:348] T 1467dfaf424441af8bfd4748a9b80c03 P c8d87365e5b945d3a3bc01249394003f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7f6033ced04c2bb63ef288745892b3" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 37247 } } peers { permanent_uuid: "072d308c8bec485381e16193e3191103" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 42451 } } peers { permanent_uuid: "c8d87365e5b945d3a3bc01249394003f" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 38575 } }
I20260929 20:21:28.812525 16304 raft_consensus.cc:374] T 1467dfaf424441af8bfd4748a9b80c03 P c8d87365e5b945d3a3bc01249394003f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:28.812654 16304 raft_consensus.cc:729] T 1467dfaf424441af8bfd4748a9b80c03 P c8d87365e5b945d3a3bc01249394003f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d87365e5b945d3a3bc01249394003f, State: Initialized, Role: FOLLOWER
I20260929 20:21:28.813267 16304 consensus_queue.cc:260] T 1467dfaf424441af8bfd4748a9b80c03 P c8d87365e5b945d3a3bc01249394003f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7f6033ced04c2bb63ef288745892b3" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 37247 } } peers { permanent_uuid: "072d308c8bec485381e16193e3191103" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 42451 } } peers { permanent_uuid: "c8d87365e5b945d3a3bc01249394003f" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 38575 } }
I20260929 20:21:28.814375 16284 heartbeater.cc:503] Master 127.27.185.126:41625 was elected leader, sending a full tablet report...
I20260929 20:21:28.815068 16304 ts_tablet_manager.cc:1428] T 1467dfaf424441af8bfd4748a9b80c03 P c8d87365e5b945d3a3bc01249394003f: Time spent starting tablet: real 0.014s user 0.009s sys 0.003s
I20260929 20:21:28.817133 16305 raft_consensus.cc:348] T 1467dfaf424441af8bfd4748a9b80c03 P 072d308c8bec485381e16193e3191103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7f6033ced04c2bb63ef288745892b3" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 37247 } } peers { permanent_uuid: "072d308c8bec485381e16193e3191103" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 42451 } } peers { permanent_uuid: "c8d87365e5b945d3a3bc01249394003f" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 38575 } }
I20260929 20:21:28.817648 16305 raft_consensus.cc:374] T 1467dfaf424441af8bfd4748a9b80c03 P 072d308c8bec485381e16193e3191103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:28.817759 16305 raft_consensus.cc:729] T 1467dfaf424441af8bfd4748a9b80c03 P 072d308c8bec485381e16193e3191103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 072d308c8bec485381e16193e3191103, State: Initialized, Role: FOLLOWER
I20260929 20:21:28.818444 16305 consensus_queue.cc:260] T 1467dfaf424441af8bfd4748a9b80c03 P 072d308c8bec485381e16193e3191103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7f6033ced04c2bb63ef288745892b3" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 37247 } } peers { permanent_uuid: "072d308c8bec485381e16193e3191103" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 42451 } } peers { permanent_uuid: "c8d87365e5b945d3a3bc01249394003f" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 38575 } }
I20260929 20:21:28.819710 16154 heartbeater.cc:503] Master 127.27.185.126:41625 was elected leader, sending a full tablet report...
I20260929 20:21:28.820271 16305 ts_tablet_manager.cc:1428] T 1467dfaf424441af8bfd4748a9b80c03 P 072d308c8bec485381e16193e3191103: Time spent starting tablet: real 0.015s user 0.008s sys 0.007s
W20260929 20:21:28.917078 16285 tablet.cc:2369] T 1467dfaf424441af8bfd4748a9b80c03 P c8d87365e5b945d3a3bc01249394003f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:21:28.939791 16308 raft_consensus.cc:482] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:21:28.940129 16308 raft_consensus.cc:504] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7f6033ced04c2bb63ef288745892b3" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 37247 } } peers { permanent_uuid: "072d308c8bec485381e16193e3191103" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 42451 } } peers { permanent_uuid: "c8d87365e5b945d3a3bc01249394003f" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 38575 } }
I20260929 20:21:28.941944 16308 leader_election.cc:288] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 072d308c8bec485381e16193e3191103 (127.27.185.66:42451), c8d87365e5b945d3a3bc01249394003f (127.27.185.67:38575)
W20260929 20:21:28.948686 16155 tablet.cc:2369] T 1467dfaf424441af8bfd4748a9b80c03 P 072d308c8bec485381e16193e3191103: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:21:28.948848 16311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:28.942380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.66:42451 (local address 127.27.185.65:39661)
0929 20:21:28.943189 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:28.943204 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:28.943330 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:28.943716 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:28.943720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:28.943744 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:21:28.944181 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:28.944192 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.945239 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.945245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:28.946787 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:28.946799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.947997 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.948003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:28.948185 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:28.948231 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 20:21:28.948300 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 20:21:28.948467 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":3,"thread_start_us":598,"threads_started":1}
I20260929 20:21:28.948913 16300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:28.942390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:39661 (local address 127.27.185.66:42451)
0929 20:21:28.942608 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:28.942617 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:28.943307 (+ 690us) server_negotiation.cc:407] Connection header received
0929 20:21:28.943471 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:28.943476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:28.943569 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:28.943672 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:21:28.944344 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.945103 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.946984 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.947812 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.947932 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:28.948023 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:28.948364 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 20:21:28.948514 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:21:28.949276 16301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:28.942471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.185.65:60185 (local address 127.27.185.67:38575)
0929 20:21:28.942736 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:28.942744 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:28.943487 (+ 743us) server_negotiation.cc:407] Connection header received
0929 20:21:28.943753 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:28.943758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:28.943841 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:28.943944 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:21:28.944540 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.945275 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.947048 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.947894 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.948060 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:28.948130 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:28.948468 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 20:21:28.948617 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 20:21:28.949525 16312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:28.942389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.67:38575 (local address 127.27.185.65:60185)
0929 20:21:28.943338 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:28.943356 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:21:28.943598 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:21:28.943966 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:21:28.943970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:21:28.943988 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:21:28.944349 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:28.944359 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.945417 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.945420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:28.946840 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:21:28.946851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.948099 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.948104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:21:28.948261 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:28.948311 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 20:21:28.949064 (+ 753us) client_negotiation.cc:240] Negotiation successful
0929 20:21:28.949206 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"mutex_wait_us":43,"thread_start_us":410,"threads_started":1}
I20260929 20:21:28.950124 16109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1467dfaf424441af8bfd4748a9b80c03" candidate_uuid: "7e7f6033ced04c2bb63ef288745892b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "072d308c8bec485381e16193e3191103" is_pre_election: true
I20260929 20:21:28.950443 16239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1467dfaf424441af8bfd4748a9b80c03" candidate_uuid: "7e7f6033ced04c2bb63ef288745892b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d87365e5b945d3a3bc01249394003f" is_pre_election: true
I20260929 20:21:28.950626 16109 raft_consensus.cc:2454] T 1467dfaf424441af8bfd4748a9b80c03 P 072d308c8bec485381e16193e3191103 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e7f6033ced04c2bb63ef288745892b3 in term 0.
I20260929 20:21:28.950963 16239 raft_consensus.cc:2454] T 1467dfaf424441af8bfd4748a9b80c03 P c8d87365e5b945d3a3bc01249394003f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e7f6033ced04c2bb63ef288745892b3 in term 0.
I20260929 20:21:28.951390 15913 leader_election.cc:302] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 072d308c8bec485381e16193e3191103, 7e7f6033ced04c2bb63ef288745892b3; no voters:
I20260929 20:21:28.951871 16308 raft_consensus.cc:2781] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:21:28.952021 16308 raft_consensus.cc:482] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:21:28.952160 16308 raft_consensus.cc:3037] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:28.954366 16308 raft_consensus.cc:504] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7f6033ced04c2bb63ef288745892b3" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 37247 } } peers { permanent_uuid: "072d308c8bec485381e16193e3191103" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 42451 } } peers { permanent_uuid: "c8d87365e5b945d3a3bc01249394003f" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 38575 } }
I20260929 20:21:28.955003 16308 leader_election.cc:288] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3 [CANDIDATE]: Term 1 election: Requested vote from peers 072d308c8bec485381e16193e3191103 (127.27.185.66:42451), c8d87365e5b945d3a3bc01249394003f (127.27.185.67:38575)
I20260929 20:21:28.955549 16239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1467dfaf424441af8bfd4748a9b80c03" candidate_uuid: "7e7f6033ced04c2bb63ef288745892b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d87365e5b945d3a3bc01249394003f"
I20260929 20:21:28.955612 16109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1467dfaf424441af8bfd4748a9b80c03" candidate_uuid: "7e7f6033ced04c2bb63ef288745892b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "072d308c8bec485381e16193e3191103"
I20260929 20:21:28.955756 16239 raft_consensus.cc:3037] T 1467dfaf424441af8bfd4748a9b80c03 P c8d87365e5b945d3a3bc01249394003f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:28.955869 16109 raft_consensus.cc:3037] T 1467dfaf424441af8bfd4748a9b80c03 P 072d308c8bec485381e16193e3191103 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:28.958082 16109 raft_consensus.cc:2454] T 1467dfaf424441af8bfd4748a9b80c03 P 072d308c8bec485381e16193e3191103 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7f6033ced04c2bb63ef288745892b3 in term 1.
I20260929 20:21:28.958134 16239 raft_consensus.cc:2454] T 1467dfaf424441af8bfd4748a9b80c03 P c8d87365e5b945d3a3bc01249394003f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7f6033ced04c2bb63ef288745892b3 in term 1.
I20260929 20:21:28.958515 15913 leader_election.cc:302] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 072d308c8bec485381e16193e3191103, 7e7f6033ced04c2bb63ef288745892b3; no voters:
I20260929 20:21:28.958846 16308 raft_consensus.cc:2781] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:21:28.959292 16308 raft_consensus.cc:686] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3 [term 1 LEADER]: Becoming Leader. State: Replica: 7e7f6033ced04c2bb63ef288745892b3, State: Running, Role: LEADER
I20260929 20:21:28.959822 16308 consensus_queue.cc:237] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7f6033ced04c2bb63ef288745892b3" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 37247 } } peers { permanent_uuid: "072d308c8bec485381e16193e3191103" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 42451 } } peers { permanent_uuid: "c8d87365e5b945d3a3bc01249394003f" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 38575 } }
I20260929 20:21:28.964831 15836 catalog_manager.cc:5495] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e7f6033ced04c2bb63ef288745892b3 (127.27.185.65). New cstate: current_term: 1 leader_uuid: "7e7f6033ced04c2bb63ef288745892b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7f6033ced04c2bb63ef288745892b3" member_type: VOTER last_known_addr { host: "127.27.185.65" port: 37247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "072d308c8bec485381e16193e3191103" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 42451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d87365e5b945d3a3bc01249394003f" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 38575 } health_report { overall_health: UNKNOWN } } }
W20260929 20:21:28.974310 16024 tablet.cc:2369] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:21:28.988346 16297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:28.982961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58730 (local address 127.27.185.65:37247)
0929 20:21:28.983193 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:28.983199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:28.983224 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:21:28.983313 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:28.983316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:28.983475 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:28.983636 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:21:28.983996 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.984700 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.986098 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:28.986510 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:28.986663 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:28.987173 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:21:28.987778 (+ 605us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:21:28.987939 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:28.987998 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 20:21:28.988082 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:21:28.996876 16109 raft_consensus.cc:1261] T 1467dfaf424441af8bfd4748a9b80c03 P 072d308c8bec485381e16193e3191103 [term 1 FOLLOWER]: Refusing update from remote peer 7e7f6033ced04c2bb63ef288745892b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:21:28.996881 16239 raft_consensus.cc:1261] T 1467dfaf424441af8bfd4748a9b80c03 P c8d87365e5b945d3a3bc01249394003f [term 1 FOLLOWER]: Refusing update from remote peer 7e7f6033ced04c2bb63ef288745892b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:21:28.997871 16313 consensus_queue.cc:1035] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d87365e5b945d3a3bc01249394003f" member_type: VOTER last_known_addr { host: "127.27.185.67" port: 38575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:21:28.998204 16308 consensus_queue.cc:1035] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "072d308c8bec485381e16193e3191103" member_type: VOTER last_known_addr { host: "127.27.185.66" port: 42451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:21:29.011590 16317 mvcc.cc:205] Tried to move back new op lower bound from 7334761631720435712 to 7334761631583588352. Current Snapshot: MvccSnapshot[applied={T|T < 7334761631720435712}]
I20260929 20:21:29.023903 15837 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:21:29.024185 15837 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:21:29.027930 15837 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0984714a40442e7b6a308058f741037: Sending DeleteTablet for 3 replicas of tablet 1467dfaf424441af8bfd4748a9b80c03
I20260929 20:21:29.029184 15958 tablet_service.cc:1514] Processing DeleteTablet for tablet 1467dfaf424441af8bfd4748a9b80c03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:21:29 UTC) from {username='slave'} at 127.0.0.1:58718
I20260929 20:21:29.029217 16089 tablet_service.cc:1514] Processing DeleteTablet for tablet 1467dfaf424441af8bfd4748a9b80c03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:21:29 UTC) from {username='slave'} at 127.0.0.1:55932
I20260929 20:21:29.029557 16219 tablet_service.cc:1514] Processing DeleteTablet for tablet 1467dfaf424441af8bfd4748a9b80c03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:21:29 UTC) from {username='slave'} at 127.0.0.1:55188
I20260929 20:21:29.030221 16326 tablet_replica.cc:330] T 1467dfaf424441af8bfd4748a9b80c03 P 072d308c8bec485381e16193e3191103: stopping tablet replica
I20260929 20:21:29.030773 16326 raft_consensus.cc:2229] T 1467dfaf424441af8bfd4748a9b80c03 P 072d308c8bec485381e16193e3191103 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:21:29.031450 16326 raft_consensus.cc:2258] T 1467dfaf424441af8bfd4748a9b80c03 P 072d308c8bec485381e16193e3191103 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:21:29.034086 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 15896
I20260929 20:21:29.043704 16326 ts_tablet_manager.cc:1905] T 1467dfaf424441af8bfd4748a9b80c03 P 072d308c8bec485381e16193e3191103: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:21:29.059262 16328 tablet_replica.cc:330] T 1467dfaf424441af8bfd4748a9b80c03 P c8d87365e5b945d3a3bc01249394003f: stopping tablet replica
I20260929 20:21:29.059777 16328 raft_consensus.cc:2229] T 1467dfaf424441af8bfd4748a9b80c03 P c8d87365e5b945d3a3bc01249394003f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:21:29.060302 16328 raft_consensus.cc:2258] T 1467dfaf424441af8bfd4748a9b80c03 P c8d87365e5b945d3a3bc01249394003f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:21:29.062790 16327 tablet_replica.cc:330] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3: stopping tablet replica
I20260929 20:21:29.062837 16328 ts_tablet_manager.cc:1905] T 1467dfaf424441af8bfd4748a9b80c03 P c8d87365e5b945d3a3bc01249394003f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:21:29.063278 16327 raft_consensus.cc:2229] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:21:29.063997 16327 raft_consensus.cc:2258] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:21:29.065465 16326 ts_tablet_manager.cc:1918] T 1467dfaf424441af8bfd4748a9b80c03 P 072d308c8bec485381e16193e3191103: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:21:29.065660 16326 log.cc:1196] T 1467dfaf424441af8bfd4748a9b80c03 P 072d308c8bec485381e16193e3191103: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-1/wal/wals/1467dfaf424441af8bfd4748a9b80c03
I20260929 20:21:29.066170 16327 ts_tablet_manager.cc:1905] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:21:29.066380 16326 ts_tablet_manager.cc:1939] T 1467dfaf424441af8bfd4748a9b80c03 P 072d308c8bec485381e16193e3191103: Deleting consensus metadata
I20260929 20:21:29.068861 15823 catalog_manager.cc:4841] TS 072d308c8bec485381e16193e3191103 (127.27.185.66:42451): tablet 1467dfaf424441af8bfd4748a9b80c03 (table test-table [id=977c0311d36f4df780c8e0cc34623939]) successfully deleted
I20260929 20:21:29.069943 16297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:29.035465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58742 (local address 127.27.185.65:37247)
0929 20:21:29.035741 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:29.035749 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:29.052137 (+ 16388us) server_negotiation.cc:407] Connection header received
0929 20:21:29.052741 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:29.052751 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:29.052893 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:29.052993 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:29.053930 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:29.055679 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:29.060944 (+ 5265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:29.061629 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:29.061771 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:29.067512 (+ 5741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:29.067525 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:29.067541 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:29.068366 (+ 825us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:29.068433 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:29.068462 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:29.068555 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:29.069395 (+ 840us) server_negotiation.cc:299] Negotiation successful
0929 20:21:29.069522 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 20:21:29.070763 16018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:29.071359 16328 ts_tablet_manager.cc:1918] T 1467dfaf424441af8bfd4748a9b80c03 P c8d87365e5b945d3a3bc01249394003f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:21:29.071766 16328 log.cc:1196] T 1467dfaf424441af8bfd4748a9b80c03 P c8d87365e5b945d3a3bc01249394003f: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-2/wal/wals/1467dfaf424441af8bfd4748a9b80c03
I20260929 20:21:29.072499 16328 ts_tablet_manager.cc:1939] T 1467dfaf424441af8bfd4748a9b80c03 P c8d87365e5b945d3a3bc01249394003f: Deleting consensus metadata
I20260929 20:21:29.075029 15822 catalog_manager.cc:4841] TS c8d87365e5b945d3a3bc01249394003f (127.27.185.67:38575): tablet 1467dfaf424441af8bfd4748a9b80c03 (table test-table [id=977c0311d36f4df780c8e0cc34623939]) successfully deleted
I20260929 20:21:29.175359 16327 ts_tablet_manager.cc:1918] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:21:29.175467 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 15896
I20260929 20:21:29.175606 16327 log.cc:1196] T 1467dfaf424441af8bfd4748a9b80c03 P 7e7f6033ced04c2bb63ef288745892b3: Deleting WAL directory at /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790712966917858-28389-0/minicluster-data/ts-0/wal/wals/1467dfaf424441af8bfd4748a9b80c03
W20260929 20:21:29.194585 15824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.185.65:37247 (error 108)
I20260929 20:21:29.195364 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 16027
I20260929 20:21:29.196777 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:29.196249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:37247 (local address 127.0.0.1:58756)
0929 20:21:29.196433 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:29.196575 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:37247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 20:21:29.197154 15824 catalog_manager.cc:4584] TS 7e7f6033ced04c2bb63ef288745892b3 (127.27.185.65:37247): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1467dfaf424441af8bfd4748a9b80c03: Network error: Client connection negotiation failed: client connection to 127.27.185.65:37247: connect: Connection refused (error 111)
I20260929 20:21:29.203445 16300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:29.196233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55936 (local address 127.27.185.66:42451)
0929 20:21:29.196516 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:29.196523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:29.196545 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:21:29.196706 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:29.196711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:29.196832 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:29.196947 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:29.197467 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:29.198265 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:29.200308 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:29.200769 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:29.200940 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:29.202053 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:29.202061 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:29.202075 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:29.202669 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:29.202716 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:29.202735 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:29.202826 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:29.202985 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 20:21:29.203073 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260929 20:21:29.204152 16149 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:29.251130 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:29.250516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:37247 (local address 127.0.0.1:58768)
0929 20:21:29.250788 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:29.250905 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:37247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260929 20:21:29.252301 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:29.251904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:37247 (local address 127.0.0.1:58784)
0929 20:21:29.252069 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:29.252155 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:37247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 20:21:29.295081 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 16027
I20260929 20:21:29.312775 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:29.312175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:37247 (local address 127.0.0.1:58794)
0929 20:21:29.312429 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:29.312555 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:37247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 20:21:29.313313 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 16157
I20260929 20:21:29.313951 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:29.313587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:37247 (local address 127.0.0.1:58806)
0929 20:21:29.313757 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:29.313835 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:37247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 20:21:29.320737 16301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:29.314139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55196 (local address 127.27.185.67:38575)
0929 20:21:29.314304 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:29.314308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:29.314487 (+ 179us) server_negotiation.cc:407] Connection header received
0929 20:21:29.314743 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:29.314748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:29.314837 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:29.314929 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:29.315282 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:29.316329 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:29.317699 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:29.318137 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:29.318258 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:29.319455 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:29.319466 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:29.319482 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:29.320082 (+ 600us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:29.320130 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:29.320150 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:29.320240 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:29.320372 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 20:21:29.320470 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 20:21:29.321437 16279 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:29.394672 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:29.394065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:37247 (local address 127.0.0.1:58814)
0929 20:21:29.394331 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:29.394452 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:37247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260929 20:21:29.395861 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:29.395376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.65:37247 (local address 127.0.0.1:58826)
0929 20:21:29.395640 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 20:21:29.395727 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.65:37247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260929 20:21:29.412606 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 16157
I20260929 20:21:29.429388 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 15806
I20260929 20:21:29.430495 15867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:29.540917 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 15806
2026-09-29T20:21:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2912 ms)
[----------] 6 tests from EncryptionPolicyTest (22222 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:21:29 dist-test-slave-4cx3 krb5kdc[16337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:21:29 dist-test-slave-4cx3 krb5kdc[16337](info): set up 2 sockets
Sep 29 20:21:29 dist-test-slave-4cx3 krb5kdc[16337](info): commencing operation
krb5kdc: starting...
W20260929 20:21:30.663142 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:21:30 dist-test-slave-4cx3 krb5kdc[16337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:21:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:21:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:21:30.926684 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36993
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:21:31.259865 16350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:31.260249 16350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:31.260427 16350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:31.267925 16350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:21:31.268097 16350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:31.268208 16350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:31.268307 16350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:21:31.268402 16350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:21:31.278630 16350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36993
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:31.280083 16350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:31.281388 16350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:31.293274 16358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:31.293367 16355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:31.293334 16356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:31.294552 16350 server_base.cc:1034] running on GCE node
I20260929 20:21:31.295454 16350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:31.296944 16350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:31.298170 16350 hybrid_clock.cc:648] HybridClock initialized: now 1790713291298147 us; error 56 us; skew 500 ppm
I20260929 20:21:31.298540 16350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:21:31 dist-test-slave-4cx3 krb5kdc[16337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:21:31.301738 16350 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:21:31.303215 16350 webserver.cc:460] Webserver started at http://127.0.0.1:45269/ using document root <none> and password file <none>
I20260929 20:21:31.303946 16350 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:31.304093 16350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:31.304405 16350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:31.307315 16350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "1d0fe49d2277468b93618546a2137363"
format_stamp: "Formatted at 2026-09-29 20:21:31 on dist-test-slave-4cx3"
server_key: "45f7ce7f206a2427db54aa81e4115873"
server_key_iv: "fd22513273ead1c9e9d8b6f1d2955afe"
server_key_version: "encryptionkey@0"
I20260929 20:21:31.308254 16350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "1d0fe49d2277468b93618546a2137363"
format_stamp: "Formatted at 2026-09-29 20:21:31 on dist-test-slave-4cx3"
server_key: "45f7ce7f206a2427db54aa81e4115873"
server_key_iv: "fd22513273ead1c9e9d8b6f1d2955afe"
server_key_version: "encryptionkey@0"
I20260929 20:21:31.313513 16350 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:21:31.317188 16365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:31.318567 16350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:21:31.318775 16350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "1d0fe49d2277468b93618546a2137363"
format_stamp: "Formatted at 2026-09-29 20:21:31 on dist-test-slave-4cx3"
server_key: "45f7ce7f206a2427db54aa81e4115873"
server_key_iv: "fd22513273ead1c9e9d8b6f1d2955afe"
server_key_version: "encryptionkey@0"
I20260929 20:21:31.318997 16350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:31.346571 16350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:31.351003 16350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:31.351318 16350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:31.370349 16350 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36993
I20260929 20:21:31.370393 16416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36993 every 8 connection(s)
I20260929 20:21:31.371955 16350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:21:31.372385 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 16350
I20260929 20:21:31.372648 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:21:31.373519 28389 external_mini_cluster.cc:1428] Setting key 6fdde4550a400e0df17e80abce3b7259
I20260929 20:21:31.376802 16417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 20:21:31 dist-test-slave-4cx3 krb5kdc[16337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 20:21:31.390707 16417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d0fe49d2277468b93618546a2137363: Bootstrap starting.
I20260929 20:21:31.395128 16417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d0fe49d2277468b93618546a2137363: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:31.396550 16417 log.cc:824] T 00000000000000000000000000000000 P 1d0fe49d2277468b93618546a2137363: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:31.399955 16420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:31.375679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58029 (local address 127.0.0.1:36993)
0929 20:21:31.376643 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:31.376654 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:31.376696 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:21:31.378430 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:31.378452 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:31.378842 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:31.379194 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:31.381012 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:31.382473 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:31.384427 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:31.385146 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:31.385276 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:31.390442 (+ 5166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:31.390454 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:31.390474 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:31.390531 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:31.394604 (+ 4073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:31.395691 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:31.395702 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:31.395711 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:31.395915 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:31.396474 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:31.396481 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:31.396487 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:31.397290 (+ 803us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:31.397596 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:31.398189 (+ 593us) server_negotiation.cc:299] Negotiation successful
0929 20:21:31.398575 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":325,"threads_started":1}
I20260929 20:21:31.400342 16417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d0fe49d2277468b93618546a2137363: No bootstrap required, opened a new log
I20260929 20:21:31.404378 28389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:21:31.408697 16417 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d0fe49d2277468b93618546a2137363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0fe49d2277468b93618546a2137363" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36993 } }
I20260929 20:21:31.409124 16417 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d0fe49d2277468b93618546a2137363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:31.409265 16417 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d0fe49d2277468b93618546a2137363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0fe49d2277468b93618546a2137363, State: Initialized, Role: FOLLOWER
I20260929 20:21:31.410007 16417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d0fe49d2277468b93618546a2137363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0fe49d2277468b93618546a2137363" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36993 } }
I20260929 20:21:31.410327 16417 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d0fe49d2277468b93618546a2137363 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:21:31.410511 16417 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d0fe49d2277468b93618546a2137363 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:21:31.410727 16417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d0fe49d2277468b93618546a2137363 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:31.413741 16417 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d0fe49d2277468b93618546a2137363 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0fe49d2277468b93618546a2137363" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36993 } }
I20260929 20:21:31.414332 16417 leader_election.cc:302] T 00000000000000000000000000000000 P 1d0fe49d2277468b93618546a2137363 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d0fe49d2277468b93618546a2137363; no voters:
I20260929 20:21:31.415019 16417 leader_election.cc:288] T 00000000000000000000000000000000 P 1d0fe49d2277468b93618546a2137363 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:21:31.415138 16427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d0fe49d2277468b93618546a2137363 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:21:31.415915 16427 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d0fe49d2277468b93618546a2137363 [term 1 LEADER]: Becoming Leader. State: Replica: 1d0fe49d2277468b93618546a2137363, State: Running, Role: LEADER
I20260929 20:21:31.416533 16427 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d0fe49d2277468b93618546a2137363 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0fe49d2277468b93618546a2137363" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36993 } }
I20260929 20:21:31.417521 16417 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d0fe49d2277468b93618546a2137363 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:21:31.419530 16429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d0fe49d2277468b93618546a2137363 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d0fe49d2277468b93618546a2137363" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0fe49d2277468b93618546a2137363" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36993 } } }
I20260929 20:21:31.419953 16429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d0fe49d2277468b93618546a2137363 [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:31.420774 16428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d0fe49d2277468b93618546a2137363 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d0fe49d2277468b93618546a2137363. Latest consensus state: current_term: 1 leader_uuid: "1d0fe49d2277468b93618546a2137363" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0fe49d2277468b93618546a2137363" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36993 } } }
I20260929 20:21:31.421109 16428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d0fe49d2277468b93618546a2137363 [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:31.425854 16432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:21:31.429201 16420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:31.409259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43667 (local address 127.0.0.1:36993)
0929 20:21:31.409488 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:31.409498 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:31.410000 (+ 502us) server_negotiation.cc:407] Connection header received
0929 20:21:31.410192 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:31.410196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:31.410257 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:31.410381 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:31.412378 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:31.413153 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:31.414954 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:31.415800 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:31.415934 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:31.419433 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:31.419444 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:31.419452 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:31.419503 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:31.424510 (+ 5007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:31.425892 (+ 1382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:31.425898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:31.425904 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:31.426113 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:31.426700 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:31.426706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:31.426712 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:31.427367 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:31.427783 (+ 416us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:31.428521 (+ 738us) server_negotiation.cc:299] Negotiation successful
0929 20:21:31.428722 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 20:21:31.433364 16432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:21:31.443768 16432 catalog_manager.cc:1347] Generated new cluster ID: cda5659cdb5a4369b2e276710b2df8b6
I20260929 20:21:31.443907 16432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:21:31.460274 16432 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:21:31.461620 16432 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:21:31.468380 16432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d0fe49d2277468b93618546a2137363: Generated new TSK 0
I20260929 20:21:31.469182 16432 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:21:31.480513 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 16350
I20260929 20:21:31.482592 16412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:31.575109 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 16350
W20260929 20:21:31.592814 16335 connection.cc:535] client connection to 127.0.0.1:36993 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36993): Connection reset by peer (error 104)
2026-09-29T20:21:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2023 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:21:31 dist-test-slave-4cx3 krb5kdc[16450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:21:31 dist-test-slave-4cx3 krb5kdc[16450](info): set up 2 sockets
Sep 29 20:21:31 dist-test-slave-4cx3 krb5kdc[16450](info): commencing operation
krb5kdc: starting...
W20260929 20:21:32.682945 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:21:32 dist-test-slave-4cx3 krb5kdc[16450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:21:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:21:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:21:32.950730 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45941
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:21:33.295359 16463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:33.295766 16463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:33.295976 16463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:33.303580 16463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:21:33.303707 16463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:33.303818 16463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:33.303915 16463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:21:33.303972 16463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:21:33.314270 16463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45941
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:33.315768 16463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:33.317173 16463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:33.329169 16468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:33.329125 16469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:33.329212 16471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:33.330190 16463 server_base.cc:1034] running on GCE node
I20260929 20:21:33.331084 16463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:33.332710 16463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:33.333920 16463 hybrid_clock.cc:648] HybridClock initialized: now 1790713293333867 us; error 44 us; skew 500 ppm
I20260929 20:21:33.334296 16463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:21:33 dist-test-slave-4cx3 krb5kdc[16450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:21:33.337646 16463 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:21:33.339222 16463 webserver.cc:460] Webserver started at http://127.0.0.1:36415/ using document root <none> and password file <none>
I20260929 20:21:33.339993 16463 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:33.340134 16463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:33.340457 16463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:33.343588 16463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "c73c47d86ed746689dfa4481065d7b16"
format_stamp: "Formatted at 2026-09-29 20:21:33 on dist-test-slave-4cx3"
server_key: "cc697b7edb05f833db3e1489529438cd"
server_key_iv: "9c16434f2c031bac324a9de3ec3b2ff0"
server_key_version: "encryptionkey@0"
I20260929 20:21:33.344389 16463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "c73c47d86ed746689dfa4481065d7b16"
format_stamp: "Formatted at 2026-09-29 20:21:33 on dist-test-slave-4cx3"
server_key: "cc697b7edb05f833db3e1489529438cd"
server_key_iv: "9c16434f2c031bac324a9de3ec3b2ff0"
server_key_version: "encryptionkey@0"
I20260929 20:21:33.349766 16463 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:21:33.353438 16478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:33.354846 16463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:21:33.355053 16463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "c73c47d86ed746689dfa4481065d7b16"
format_stamp: "Formatted at 2026-09-29 20:21:33 on dist-test-slave-4cx3"
server_key: "cc697b7edb05f833db3e1489529438cd"
server_key_iv: "9c16434f2c031bac324a9de3ec3b2ff0"
server_key_version: "encryptionkey@0"
I20260929 20:21:33.355283 16463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:33.378945 16463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:33.383311 16463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:33.383665 16463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:33.402801 16463 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45941
I20260929 20:21:33.402832 16529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45941 every 8 connection(s)
I20260929 20:21:33.404251 16463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:21:33.406904 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 16463
I20260929 20:21:33.407102 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:21:33.407687 28389 external_mini_cluster.cc:1428] Setting key e6435154f12fd219f1143ea378be12e7
I20260929 20:21:33.408639 16530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:33.420881 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:33.409520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47121 (local address 127.0.0.1:45941)
0929 20:21:33.410351 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:33.410363 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:33.410405 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:21:33.412131 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:33.412154 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:33.412575 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:33.413004 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:33.413564 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:33.414909 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:33.416328 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:33.416997 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:33.417195 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:33.418523 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:33.418537 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:33.418554 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:33.419146 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:33.419227 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:33.419255 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:33.419465 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:33.419736 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 20:21:33.419914 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":346,"threads_started":1}
I20260929 20:21:33.421447 16530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c73c47d86ed746689dfa4481065d7b16: Bootstrap starting.
I20260929 20:21:33.424881 28389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:21:33.424907 16530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c73c47d86ed746689dfa4481065d7b16: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:33.425964 16530 log.cc:824] T 00000000000000000000000000000000 P c73c47d86ed746689dfa4481065d7b16: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:33.429641 16530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c73c47d86ed746689dfa4481065d7b16: No bootstrap required, opened a new log
I20260929 20:21:33.436424 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:33.429474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57685 (local address 127.0.0.1:45941)
0929 20:21:33.429669 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:33.429675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:33.430042 (+ 367us) server_negotiation.cc:407] Connection header received
0929 20:21:33.430229 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:33.430234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:33.430319 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:33.430441 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:33.430925 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:33.432130 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:33.433534 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:33.434234 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:33.434363 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:33.435535 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:33.435543 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:33.435553 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:33.435579 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:33.435647 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:33.435669 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:33.435789 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:33.435992 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 20:21:33.436053 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:21:33.439736 16530 raft_consensus.cc:348] T 00000000000000000000000000000000 P c73c47d86ed746689dfa4481065d7b16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73c47d86ed746689dfa4481065d7b16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45941 } }
I20260929 20:21:33.440127 16530 raft_consensus.cc:374] T 00000000000000000000000000000000 P c73c47d86ed746689dfa4481065d7b16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:33.440207 16530 raft_consensus.cc:729] T 00000000000000000000000000000000 P c73c47d86ed746689dfa4481065d7b16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73c47d86ed746689dfa4481065d7b16, State: Initialized, Role: FOLLOWER
I20260929 20:21:33.440749 16530 consensus_queue.cc:260] T 00000000000000000000000000000000 P c73c47d86ed746689dfa4481065d7b16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73c47d86ed746689dfa4481065d7b16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45941 } }
I20260929 20:21:33.440964 16530 raft_consensus.cc:388] T 00000000000000000000000000000000 P c73c47d86ed746689dfa4481065d7b16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:21:33.441073 16530 raft_consensus.cc:482] T 00000000000000000000000000000000 P c73c47d86ed746689dfa4481065d7b16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:21:33.441207 16530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c73c47d86ed746689dfa4481065d7b16 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:33.443360 16530 raft_consensus.cc:504] T 00000000000000000000000000000000 P c73c47d86ed746689dfa4481065d7b16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73c47d86ed746689dfa4481065d7b16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45941 } }
I20260929 20:21:33.443971 16530 leader_election.cc:302] T 00000000000000000000000000000000 P c73c47d86ed746689dfa4481065d7b16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c73c47d86ed746689dfa4481065d7b16; no voters:
I20260929 20:21:33.444767 16530 leader_election.cc:288] T 00000000000000000000000000000000 P c73c47d86ed746689dfa4481065d7b16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:21:33.444881 16540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c73c47d86ed746689dfa4481065d7b16 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:21:33.445710 16540 raft_consensus.cc:686] T 00000000000000000000000000000000 P c73c47d86ed746689dfa4481065d7b16 [term 1 LEADER]: Becoming Leader. State: Replica: c73c47d86ed746689dfa4481065d7b16, State: Running, Role: LEADER
I20260929 20:21:33.446308 16540 consensus_queue.cc:237] T 00000000000000000000000000000000 P c73c47d86ed746689dfa4481065d7b16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73c47d86ed746689dfa4481065d7b16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45941 } }
I20260929 20:21:33.447007 16530 sys_catalog.cc:564] T 00000000000000000000000000000000 P c73c47d86ed746689dfa4481065d7b16 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:21:33.449577 16541 sys_catalog.cc:455] T 00000000000000000000000000000000 P c73c47d86ed746689dfa4481065d7b16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c73c47d86ed746689dfa4481065d7b16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73c47d86ed746689dfa4481065d7b16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45941 } } }
I20260929 20:21:33.449973 16541 sys_catalog.cc:458] T 00000000000000000000000000000000 P c73c47d86ed746689dfa4481065d7b16 [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:33.452243 16542 sys_catalog.cc:455] T 00000000000000000000000000000000 P c73c47d86ed746689dfa4481065d7b16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c73c47d86ed746689dfa4481065d7b16. Latest consensus state: current_term: 1 leader_uuid: "c73c47d86ed746689dfa4481065d7b16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73c47d86ed746689dfa4481065d7b16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45941 } } }
I20260929 20:21:33.452664 16542 sys_catalog.cc:458] T 00000000000000000000000000000000 P c73c47d86ed746689dfa4481065d7b16 [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:33.453809 16546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:21:33.460742 16546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:21:33.470249 16546 catalog_manager.cc:1347] Generated new cluster ID: 0de91c7d77504b60b88e0a1a6c0fd40e
I20260929 20:21:33.470386 16546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:21:33.488207 16546 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:21:33.489555 16546 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:21:33.497699 16546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c73c47d86ed746689dfa4481065d7b16: Generated new TSK 0
I20260929 20:21:33.498835 16546 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:21:33.514731 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 16463
W20260929 20:21:33.515965 16525 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47121
W20260929 20:21:33.516752 28389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 16463: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:21:33.516952 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 16463
W20260929 20:21:33.534683 16448 connection.cc:535] client connection to 127.0.0.1:45941 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45941): Connection reset by peer (error 104)
2026-09-29T20:21:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1941 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:21:33 dist-test-slave-4cx3 krb5kdc[16561](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:21:33 dist-test-slave-4cx3 krb5kdc[16561](info): set up 2 sockets
Sep 29 20:21:33 dist-test-slave-4cx3 krb5kdc[16561](info): commencing operation
krb5kdc: starting...
W20260929 20:21:34.650846 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:21:34 dist-test-slave-4cx3 krb5kdc[16561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:21:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:21:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:21:34.911456 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37043
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:21:35.246379 16574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:35.246685 16574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:35.246778 16574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:35.254184 16574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:21:35.254307 16574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:35.254355 16574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:35.254428 16574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:21:35.254473 16574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:21:35.264912 16574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37043
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:35.266197 16574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:35.267601 16574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:35.279556 16580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:35.279584 16582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:35.279609 16579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:35.280495 16574 server_base.cc:1034] running on GCE node
I20260929 20:21:35.281352 16574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:35.282829 16574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:35.284035 16574 hybrid_clock.cc:648] HybridClock initialized: now 1790713295283979 us; error 50 us; skew 500 ppm
I20260929 20:21:35.284435 16574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:21:35 dist-test-slave-4cx3 krb5kdc[16561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:21:35.293107 16574 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:21:35.294896 16574 webserver.cc:460] Webserver started at http://127.0.0.1:45461/ using document root <none> and password file <none>
I20260929 20:21:35.295708 16574 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:35.295853 16574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:35.296185 16574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:35.299265 16574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "59fead027c9e4844bcc54f7ec92d0b3c"
format_stamp: "Formatted at 2026-09-29 20:21:35 on dist-test-slave-4cx3"
server_key: "2188f37e513ef1808ae0ce4983e2b6ee"
server_key_iv: "39c373681c7e65db9f01cbb6c2472eed"
server_key_version: "encryptionkey@0"
I20260929 20:21:35.300129 16574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "59fead027c9e4844bcc54f7ec92d0b3c"
format_stamp: "Formatted at 2026-09-29 20:21:35 on dist-test-slave-4cx3"
server_key: "2188f37e513ef1808ae0ce4983e2b6ee"
server_key_iv: "39c373681c7e65db9f01cbb6c2472eed"
server_key_version: "encryptionkey@0"
I20260929 20:21:35.305475 16574 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:21:35.309021 16589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:35.310415 16574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:21:35.310604 16574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "59fead027c9e4844bcc54f7ec92d0b3c"
format_stamp: "Formatted at 2026-09-29 20:21:35 on dist-test-slave-4cx3"
server_key: "2188f37e513ef1808ae0ce4983e2b6ee"
server_key_iv: "39c373681c7e65db9f01cbb6c2472eed"
server_key_version: "encryptionkey@0"
I20260929 20:21:35.310818 16574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:35.316772 16574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:35.320657 16574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:35.320946 16574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:35.339761 16574 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37043
I20260929 20:21:35.339809 16640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37043 every 8 connection(s)
I20260929 20:21:35.341290 16574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:21:35.345031 16641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:35.345877 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 16574
I20260929 20:21:35.346088 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:21:35.346565 28389 external_mini_cluster.cc:1428] Setting key 0ba2d9547b14dbaaa0cae463a9c89cc4
I20260929 20:21:35.355487 16644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:35.348468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40703 (local address 127.0.0.1:37043)
0929 20:21:35.349260 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:35.349277 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:35.349349 (+ 72us) server_negotiation.cc:407] Connection header received
0929 20:21:35.351413 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:35.351445 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:35.351947 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:35.352445 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:35.353503 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:35.353513 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:35.353530 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:35.353558 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:35.353631 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:35.353659 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:35.353894 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:35.354019 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 20:21:35.354217 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":303,"threads_started":1}
I20260929 20:21:35.359463 28389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:21:35.360374 16641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59fead027c9e4844bcc54f7ec92d0b3c: Bootstrap starting.
I20260929 20:21:35.364578 16641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59fead027c9e4844bcc54f7ec92d0b3c: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:35.366073 16641 log.cc:824] T 00000000000000000000000000000000 P 59fead027c9e4844bcc54f7ec92d0b3c: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:35.367561 16644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:35.364917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51945 (local address 127.0.0.1:37043)
0929 20:21:35.365136 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:35.365144 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:35.365450 (+ 306us) server_negotiation.cc:407] Connection header received
0929 20:21:35.365602 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:35.365610 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:35.365677 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:35.365808 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:35.366822 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:35.366826 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:35.366829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:35.366844 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:35.366882 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:35.366896 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:35.366993 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:35.367161 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 20:21:35.367213 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 20:21:35.369375 16641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59fead027c9e4844bcc54f7ec92d0b3c: No bootstrap required, opened a new log
I20260929 20:21:35.377249 16641 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59fead027c9e4844bcc54f7ec92d0b3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fead027c9e4844bcc54f7ec92d0b3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37043 } }
I20260929 20:21:35.377686 16641 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59fead027c9e4844bcc54f7ec92d0b3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:35.377799 16641 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59fead027c9e4844bcc54f7ec92d0b3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59fead027c9e4844bcc54f7ec92d0b3c, State: Initialized, Role: FOLLOWER
I20260929 20:21:35.378507 16641 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59fead027c9e4844bcc54f7ec92d0b3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fead027c9e4844bcc54f7ec92d0b3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37043 } }
I20260929 20:21:35.378758 16641 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59fead027c9e4844bcc54f7ec92d0b3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:21:35.378913 16641 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59fead027c9e4844bcc54f7ec92d0b3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:21:35.379099 16641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59fead027c9e4844bcc54f7ec92d0b3c [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:35.381853 16641 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59fead027c9e4844bcc54f7ec92d0b3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fead027c9e4844bcc54f7ec92d0b3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37043 } }
I20260929 20:21:35.382407 16641 leader_election.cc:302] T 00000000000000000000000000000000 P 59fead027c9e4844bcc54f7ec92d0b3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59fead027c9e4844bcc54f7ec92d0b3c; no voters:
I20260929 20:21:35.383001 16641 leader_election.cc:288] T 00000000000000000000000000000000 P 59fead027c9e4844bcc54f7ec92d0b3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:21:35.383173 16651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59fead027c9e4844bcc54f7ec92d0b3c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:21:35.383900 16651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59fead027c9e4844bcc54f7ec92d0b3c [term 1 LEADER]: Becoming Leader. State: Replica: 59fead027c9e4844bcc54f7ec92d0b3c, State: Running, Role: LEADER
I20260929 20:21:35.384464 16651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59fead027c9e4844bcc54f7ec92d0b3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fead027c9e4844bcc54f7ec92d0b3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37043 } }
I20260929 20:21:35.385212 16641 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59fead027c9e4844bcc54f7ec92d0b3c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:21:35.387789 16653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59fead027c9e4844bcc54f7ec92d0b3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59fead027c9e4844bcc54f7ec92d0b3c. Latest consensus state: current_term: 1 leader_uuid: "59fead027c9e4844bcc54f7ec92d0b3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fead027c9e4844bcc54f7ec92d0b3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37043 } } }
I20260929 20:21:35.388188 16653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59fead027c9e4844bcc54f7ec92d0b3c [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:35.388602 16652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59fead027c9e4844bcc54f7ec92d0b3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59fead027c9e4844bcc54f7ec92d0b3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fead027c9e4844bcc54f7ec92d0b3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37043 } } }
I20260929 20:21:35.388885 16652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59fead027c9e4844bcc54f7ec92d0b3c [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:35.396570 16659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:21:35.402221 16659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:21:35.410959 16659 catalog_manager.cc:1347] Generated new cluster ID: 5d1f16cc93ac4b43b8995ae38cb86436
I20260929 20:21:35.411105 16659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:21:35.427237 16659 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:21:35.427719 16659 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:21:35.442587 16659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59fead027c9e4844bcc54f7ec92d0b3c: Generated new TSK 0
I20260929 20:21:35.443768 16659 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:21:35.452939 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 16574
W20260929 20:21:35.454845 16636 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40703
W20260929 20:21:35.455687 28389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 16574: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:21:35.455844 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 16574
2026-09-29T20:21:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1935 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:21:35 dist-test-slave-4cx3 krb5kdc[16672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:21:35 dist-test-slave-4cx3 krb5kdc[16672](info): set up 2 sockets
Sep 29 20:21:35 dist-test-slave-4cx3 krb5kdc[16672](info): commencing operation
krb5kdc: starting...
W20260929 20:21:36.566747 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:21:36 dist-test-slave-4cx3 krb5kdc[16672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:21:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:21:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:21:36.835434 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36159
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:21:37.173810 16685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:37.174120 16685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:37.174278 16685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:37.181866 16685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:21:37.181998 16685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:37.182085 16685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:37.182202 16685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:21:37.182272 16685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:21:37.193022 16685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:37.194446 16685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:37.195953 16685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:37.208053 16691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:37.208269 16693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:37.208120 16690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:37.209280 16685 server_base.cc:1034] running on GCE node
I20260929 20:21:37.210163 16685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:37.211719 16685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:37.212953 16685 hybrid_clock.cc:648] HybridClock initialized: now 1790713297212905 us; error 61 us; skew 500 ppm
I20260929 20:21:37.213331 16685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:21:37 dist-test-slave-4cx3 krb5kdc[16672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:21:37.216763 16685 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:21:37.218290 16685 webserver.cc:460] Webserver started at http://127.0.0.1:34197/ using document root <none> and password file <none>
I20260929 20:21:37.219024 16685 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:37.219178 16685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:37.219560 16685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:37.222703 16685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "3fbf54b85069464ea2411a279eba84b6"
format_stamp: "Formatted at 2026-09-29 20:21:37 on dist-test-slave-4cx3"
server_key: "c6fd9c5bf0104bec3b1ad8c4ae51d354"
server_key_iv: "64316abb6cf11cf472a86b05ba0e66c6"
server_key_version: "encryptionkey@0"
I20260929 20:21:37.223721 16685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "3fbf54b85069464ea2411a279eba84b6"
format_stamp: "Formatted at 2026-09-29 20:21:37 on dist-test-slave-4cx3"
server_key: "c6fd9c5bf0104bec3b1ad8c4ae51d354"
server_key_iv: "64316abb6cf11cf472a86b05ba0e66c6"
server_key_version: "encryptionkey@0"
I20260929 20:21:37.229131 16685 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:21:37.233196 16700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:37.234760 16685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 20:21:37.234972 16685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "3fbf54b85069464ea2411a279eba84b6"
format_stamp: "Formatted at 2026-09-29 20:21:37 on dist-test-slave-4cx3"
server_key: "c6fd9c5bf0104bec3b1ad8c4ae51d354"
server_key_iv: "64316abb6cf11cf472a86b05ba0e66c6"
server_key_version: "encryptionkey@0"
I20260929 20:21:37.235201 16685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:37.256876 16685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:37.261586 16685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:37.261898 16685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:37.281399 16685 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36159
I20260929 20:21:37.281411 16751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36159 every 8 connection(s)
I20260929 20:21:37.282883 16685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:21:37.286903 16752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:37.291272 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 16685
I20260929 20:21:37.291498 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:21:37.291925 28389 external_mini_cluster.cc:1428] Setting key ecd7b671da3a61c61130f2ee847bf97e
I20260929 20:21:37.298357 16752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fbf54b85069464ea2411a279eba84b6: Bootstrap starting.
I20260929 20:21:37.302181 16752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fbf54b85069464ea2411a279eba84b6: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:37.303483 16752 log.cc:824] T 00000000000000000000000000000000 P 3fbf54b85069464ea2411a279eba84b6: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:37.307219 16752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fbf54b85069464ea2411a279eba84b6: No bootstrap required, opened a new log
Sep 29 20:21:37 dist-test-slave-4cx3 krb5kdc[16672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 20:21:37.315341 16752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fbf54b85069464ea2411a279eba84b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fbf54b85069464ea2411a279eba84b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36159 } }
I20260929 20:21:37.315838 16752 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fbf54b85069464ea2411a279eba84b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:37.315991 16752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fbf54b85069464ea2411a279eba84b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fbf54b85069464ea2411a279eba84b6, State: Initialized, Role: FOLLOWER
I20260929 20:21:37.316598 16752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fbf54b85069464ea2411a279eba84b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fbf54b85069464ea2411a279eba84b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36159 } }
I20260929 20:21:37.316890 16752 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fbf54b85069464ea2411a279eba84b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:21:37.317049 16752 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fbf54b85069464ea2411a279eba84b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:21:37.317246 16752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fbf54b85069464ea2411a279eba84b6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:37.317581 16755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:37.293736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59223 (local address 127.0.0.1:36159)
0929 20:21:37.294524 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:37.294536 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:37.294577 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:21:37.296850 (+ 2273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:37.296876 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:37.297313 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:37.297656 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:37.299654 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:37.301167 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:37.302992 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:37.303966 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:37.304223 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:37.309172 (+ 4949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:37.309180 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:37.309196 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:37.309244 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:37.312657 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:37.313441 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:37.313452 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:37.313459 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:37.313650 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:37.314075 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:37.314081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:37.314094 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:37.314683 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:37.315008 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:37.315514 (+ 506us) server_negotiation.cc:299] Negotiation successful
0929 20:21:37.316219 (+ 705us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":360,"threads_started":1}
I20260929 20:21:37.320473 16752 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fbf54b85069464ea2411a279eba84b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fbf54b85069464ea2411a279eba84b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36159 } }
I20260929 20:21:37.321100 16752 leader_election.cc:302] T 00000000000000000000000000000000 P 3fbf54b85069464ea2411a279eba84b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fbf54b85069464ea2411a279eba84b6; no voters:
I20260929 20:21:37.321738 28389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:21:37.321851 16752 leader_election.cc:288] T 00000000000000000000000000000000 P 3fbf54b85069464ea2411a279eba84b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:21:37.321961 16757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fbf54b85069464ea2411a279eba84b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:21:37.324045 16752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fbf54b85069464ea2411a279eba84b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:21:37.332129 16757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fbf54b85069464ea2411a279eba84b6 [term 1 LEADER]: Becoming Leader. State: Replica: 3fbf54b85069464ea2411a279eba84b6, State: Running, Role: LEADER
I20260929 20:21:37.333047 16757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fbf54b85069464ea2411a279eba84b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fbf54b85069464ea2411a279eba84b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36159 } }
I20260929 20:21:37.336875 16759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fbf54b85069464ea2411a279eba84b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fbf54b85069464ea2411a279eba84b6. Latest consensus state: current_term: 1 leader_uuid: "3fbf54b85069464ea2411a279eba84b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fbf54b85069464ea2411a279eba84b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36159 } } }
I20260929 20:21:37.337325 16759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fbf54b85069464ea2411a279eba84b6 [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:37.337916 16758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fbf54b85069464ea2411a279eba84b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fbf54b85069464ea2411a279eba84b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fbf54b85069464ea2411a279eba84b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36159 } } }
I20260929 20:21:37.338194 16758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fbf54b85069464ea2411a279eba84b6 [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:37.349234 16774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:21:37.355509 16774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:21:37.356595 16755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:37.334664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48489 (local address 127.0.0.1:36159)
0929 20:21:37.334845 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:37.334851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:37.335642 (+ 791us) server_negotiation.cc:407] Connection header received
0929 20:21:37.335959 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:37.335966 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:37.336058 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:37.336139 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:37.338554 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:37.340030 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:37.342433 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:37.343379 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:37.343537 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:37.346815 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:37.346823 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:37.346828 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:37.346875 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:37.350794 (+ 3919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:37.352040 (+ 1246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:37.352054 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:37.352060 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:37.352260 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:37.352849 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:37.352857 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:37.352864 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:37.353457 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:37.353758 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:37.355590 (+ 1832us) server_negotiation.cc:299] Negotiation successful
0929 20:21:37.355916 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:21:37.367065 16774 catalog_manager.cc:1347] Generated new cluster ID: 34641ddda7e540f681fd9bdd9baf2bc0
I20260929 20:21:37.367201 16774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:21:37.419013 16774 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:21:37.421021 16774 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:21:37.430171 16774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fbf54b85069464ea2411a279eba84b6: Generated new TSK 0
I20260929 20:21:37.431070 16774 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:21:37.442761 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 16685
I20260929 20:21:37.444169 16747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:37.540446 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 16685
2026-09-29T20:21:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2088 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:21:37 dist-test-slave-4cx3 krb5kdc[16784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:21:37 dist-test-slave-4cx3 krb5kdc[16784](info): set up 2 sockets
Sep 29 20:21:37 dist-test-slave-4cx3 krb5kdc[16784](info): commencing operation
krb5kdc: starting...
W20260929 20:21:38.655061 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:21:38 dist-test-slave-4cx3 krb5kdc[16784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:21:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:21:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:21:38.918367 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35537
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:21:39.255966 16797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:39.256278 16797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:39.256495 16797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:39.263811 16797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:21:39.263957 16797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:39.264045 16797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:39.264159 16797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:21:39.264246 16797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:21:39.274307 16797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:39.275802 16797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:39.277233 16797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:39.289283 16803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:39.289368 16805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:39.289391 16802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:39.290230 16797 server_base.cc:1034] running on GCE node
I20260929 20:21:39.291123 16797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:39.292626 16797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:39.293829 16797 hybrid_clock.cc:648] HybridClock initialized: now 1790713299293784 us; error 36 us; skew 500 ppm
I20260929 20:21:39.294193 16797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:21:39 dist-test-slave-4cx3 krb5kdc[16784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:21:39.297719 16797 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:21:39.299146 16797 webserver.cc:460] Webserver started at http://127.0.0.1:44141/ using document root <none> and password file <none>
I20260929 20:21:39.299868 16797 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:39.299971 16797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:39.300215 16797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:39.303160 16797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "dfcbd3bd2e774e3594dfc682a0dd9351"
format_stamp: "Formatted at 2026-09-29 20:21:39 on dist-test-slave-4cx3"
server_key: "9268cf8f32f804c7698be8305f44d74c"
server_key_iv: "63660162cd80b2b7261dd5e2df42ab42"
server_key_version: "encryptionkey@0"
I20260929 20:21:39.303932 16797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "dfcbd3bd2e774e3594dfc682a0dd9351"
format_stamp: "Formatted at 2026-09-29 20:21:39 on dist-test-slave-4cx3"
server_key: "9268cf8f32f804c7698be8305f44d74c"
server_key_iv: "63660162cd80b2b7261dd5e2df42ab42"
server_key_version: "encryptionkey@0"
I20260929 20:21:39.308980 16797 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:21:39.312430 16812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:39.313813 16797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:21:39.314007 16797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "dfcbd3bd2e774e3594dfc682a0dd9351"
format_stamp: "Formatted at 2026-09-29 20:21:39 on dist-test-slave-4cx3"
server_key: "9268cf8f32f804c7698be8305f44d74c"
server_key_iv: "63660162cd80b2b7261dd5e2df42ab42"
server_key_version: "encryptionkey@0"
I20260929 20:21:39.314214 16797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:39.333849 16797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:39.338523 16797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:39.338824 16797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:39.357741 16797 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35537
I20260929 20:21:39.357782 16863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35537 every 8 connection(s)
I20260929 20:21:39.359223 16797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:21:39.363193 16864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:39.363847 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 16797
I20260929 20:21:39.364123 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:21:39.364689 28389 external_mini_cluster.cc:1428] Setting key b842e5a518d22eed43a1c21a756efd66
I20260929 20:21:39.378623 16864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfcbd3bd2e774e3594dfc682a0dd9351: Bootstrap starting.
I20260929 20:21:39.380834 16867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:39.366628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59559 (local address 127.0.0.1:35537)
0929 20:21:39.367497 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:39.367513 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:39.367579 (+ 66us) server_negotiation.cc:407] Connection header received
0929 20:21:39.370231 (+ 2652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:39.370265 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:39.370874 (+ 609us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:39.371368 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:39.371949 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:39.373827 (+ 1878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:39.375242 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:39.376445 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:39.376641 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:39.377867 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:39.377878 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:39.377897 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:39.378659 (+ 762us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:39.378762 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:39.378793 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:39.378914 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:39.379140 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 20:21:39.379442 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":345,"threads_started":1}
I20260929 20:21:39.382438 16864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfcbd3bd2e774e3594dfc682a0dd9351: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:39.383622 16864 log.cc:824] T 00000000000000000000000000000000 P dfcbd3bd2e774e3594dfc682a0dd9351: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:39.385032 28389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:21:39.387009 16864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfcbd3bd2e774e3594dfc682a0dd9351: No bootstrap required, opened a new log
I20260929 20:21:39.397495 16867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:39.390424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57531 (local address 127.0.0.1:35537)
0929 20:21:39.390699 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:39.390709 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:39.391247 (+ 538us) server_negotiation.cc:407] Connection header received
0929 20:21:39.391416 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:39.391422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:39.391508 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:39.391633 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:39.392141 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:39.393318 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:39.394758 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:39.395518 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:39.395677 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:39.396725 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:39.396731 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:39.396735 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:39.396759 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:39.396804 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:39.396822 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:39.396917 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:39.397058 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 20:21:39.397151 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:21:39.397198 16864 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfcbd3bd2e774e3594dfc682a0dd9351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcbd3bd2e774e3594dfc682a0dd9351" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35537 } }
I20260929 20:21:39.397723 16864 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfcbd3bd2e774e3594dfc682a0dd9351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:39.397864 16864 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfcbd3bd2e774e3594dfc682a0dd9351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfcbd3bd2e774e3594dfc682a0dd9351, State: Initialized, Role: FOLLOWER
I20260929 20:21:39.398595 16864 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfcbd3bd2e774e3594dfc682a0dd9351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcbd3bd2e774e3594dfc682a0dd9351" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35537 } }
I20260929 20:21:39.398900 16864 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfcbd3bd2e774e3594dfc682a0dd9351 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:21:39.399070 16864 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfcbd3bd2e774e3594dfc682a0dd9351 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:21:39.399260 16864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfcbd3bd2e774e3594dfc682a0dd9351 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:39.401481 16864 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfcbd3bd2e774e3594dfc682a0dd9351 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcbd3bd2e774e3594dfc682a0dd9351" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35537 } }
I20260929 20:21:39.402006 16864 leader_election.cc:302] T 00000000000000000000000000000000 P dfcbd3bd2e774e3594dfc682a0dd9351 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfcbd3bd2e774e3594dfc682a0dd9351; no voters:
I20260929 20:21:39.402662 16864 leader_election.cc:288] T 00000000000000000000000000000000 P dfcbd3bd2e774e3594dfc682a0dd9351 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:21:39.402823 16874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfcbd3bd2e774e3594dfc682a0dd9351 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:21:39.403563 16874 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfcbd3bd2e774e3594dfc682a0dd9351 [term 1 LEADER]: Becoming Leader. State: Replica: dfcbd3bd2e774e3594dfc682a0dd9351, State: Running, Role: LEADER
I20260929 20:21:39.404603 16874 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfcbd3bd2e774e3594dfc682a0dd9351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcbd3bd2e774e3594dfc682a0dd9351" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35537 } }
I20260929 20:21:39.405097 16864 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfcbd3bd2e774e3594dfc682a0dd9351 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:21:39.407665 16876 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfcbd3bd2e774e3594dfc682a0dd9351 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfcbd3bd2e774e3594dfc682a0dd9351. Latest consensus state: current_term: 1 leader_uuid: "dfcbd3bd2e774e3594dfc682a0dd9351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcbd3bd2e774e3594dfc682a0dd9351" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35537 } } }
I20260929 20:21:39.407694 16875 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfcbd3bd2e774e3594dfc682a0dd9351 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfcbd3bd2e774e3594dfc682a0dd9351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcbd3bd2e774e3594dfc682a0dd9351" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35537 } } }
I20260929 20:21:39.408107 16875 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfcbd3bd2e774e3594dfc682a0dd9351 [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:39.408107 16876 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfcbd3bd2e774e3594dfc682a0dd9351 [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:39.409760 16881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:21:39.414951 16881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:21:39.427409 16881 catalog_manager.cc:1347] Generated new cluster ID: d9675e794d21497d92b045271e911c85
I20260929 20:21:39.427639 16881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:21:39.452961 16881 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:21:39.454871 16881 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:21:39.470871 16881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfcbd3bd2e774e3594dfc682a0dd9351: Generated new TSK 0
I20260929 20:21:39.471976 16881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:21:39.479323 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 16797
W20260929 20:21:39.480522 16859 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59559
W20260929 20:21:39.481240 28389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 16797: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:21:39.481391 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 16797
2026-09-29T20:21:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1940 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:21:39 dist-test-slave-4cx3 krb5kdc[16895](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:21:39 dist-test-slave-4cx3 krb5kdc[16895](info): set up 2 sockets
Sep 29 20:21:39 dist-test-slave-4cx3 krb5kdc[16895](info): commencing operation
krb5kdc: starting...
W20260929 20:21:40.602923 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:21:40 dist-test-slave-4cx3 krb5kdc[16895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:21:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:21:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:21:40.864622 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41749
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:21:41.201079 16908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:41.201382 16908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:41.201576 16908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:41.208900 16908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:21:41.209039 16908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:41.209146 16908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:41.209245 16908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:21:41.209343 16908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:21:41.219521 16908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:41.220884 16908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:41.222239 16908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:41.234444 16913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:41.234503 16914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:41.234407 16916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:41.235478 16908 server_base.cc:1034] running on GCE node
I20260929 20:21:41.236339 16908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:41.237834 16908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:41.239061 16908 hybrid_clock.cc:648] HybridClock initialized: now 1790713301239011 us; error 39 us; skew 500 ppm
I20260929 20:21:41.239540 16908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:21:41 dist-test-slave-4cx3 krb5kdc[16895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:21:41.249722 16908 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:21:41.251451 16908 webserver.cc:460] Webserver started at http://127.0.0.1:41515/ using document root <none> and password file <none>
I20260929 20:21:41.252192 16908 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:41.252342 16908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:41.252666 16908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:41.255880 16908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "ae91a1e97a5f49a9a70bc3d90d12d07c"
format_stamp: "Formatted at 2026-09-29 20:21:41 on dist-test-slave-4cx3"
server_key: "142df4866006f28d14319a7dcc16653c"
server_key_iv: "f22bfc4feca735c131bd7e5c7d474c56"
server_key_version: "encryptionkey@0"
I20260929 20:21:41.256705 16908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "ae91a1e97a5f49a9a70bc3d90d12d07c"
format_stamp: "Formatted at 2026-09-29 20:21:41 on dist-test-slave-4cx3"
server_key: "142df4866006f28d14319a7dcc16653c"
server_key_iv: "f22bfc4feca735c131bd7e5c7d474c56"
server_key_version: "encryptionkey@0"
I20260929 20:21:41.262136 16908 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 20:21:41.265699 16923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:41.266988 16908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 20:21:41.267180 16908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "ae91a1e97a5f49a9a70bc3d90d12d07c"
format_stamp: "Formatted at 2026-09-29 20:21:41 on dist-test-slave-4cx3"
server_key: "142df4866006f28d14319a7dcc16653c"
server_key_iv: "f22bfc4feca735c131bd7e5c7d474c56"
server_key_version: "encryptionkey@0"
I20260929 20:21:41.267388 16908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:41.273538 16908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:41.277623 16908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:41.277920 16908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:41.296911 16908 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41749
I20260929 20:21:41.296957 16974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41749 every 8 connection(s)
I20260929 20:21:41.298398 16908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:21:41.300014 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 16908
I20260929 20:21:41.300190 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:21:41.300734 28389 external_mini_cluster.cc:1428] Setting key 3e07deac4a2cd8a73e1bb057e63c4f16
I20260929 20:21:41.303478 16975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:41.308637 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:41.302579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32775 (local address 127.0.0.1:41749)
0929 20:21:41.303472 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:41.303485 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:41.303530 (+ 45us) server_negotiation.cc:407] Connection header received
0929 20:21:41.305186 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:41.305208 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:41.305549 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:41.305887 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:41.306961 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:41.306967 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:41.306980 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:41.306994 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:41.307067 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:41.307086 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:41.307272 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:41.307375 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 20:21:41.307694 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":368,"threads_started":1}
I20260929 20:21:41.312441 28389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:21:41.316498 16975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae91a1e97a5f49a9a70bc3d90d12d07c: Bootstrap starting.
I20260929 20:21:41.320336 16975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae91a1e97a5f49a9a70bc3d90d12d07c: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:41.320552 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:41.317797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57051 (local address 127.0.0.1:41749)
0929 20:21:41.318024 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:41.318033 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:41.318552 (+ 519us) server_negotiation.cc:407] Connection header received
0929 20:21:41.318700 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:41.318705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:41.318766 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:41.318877 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:41.319794 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:41.319799 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:41.319804 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:41.319821 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:41.319871 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:41.319891 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:41.320008 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:41.320114 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 20:21:41.320177 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 20:21:41.321642 16975 log.cc:824] T 00000000000000000000000000000000 P ae91a1e97a5f49a9a70bc3d90d12d07c: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:41.324780 16975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae91a1e97a5f49a9a70bc3d90d12d07c: No bootstrap required, opened a new log
I20260929 20:21:41.332280 16975 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae91a1e97a5f49a9a70bc3d90d12d07c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae91a1e97a5f49a9a70bc3d90d12d07c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41749 } }
I20260929 20:21:41.332721 16975 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae91a1e97a5f49a9a70bc3d90d12d07c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:41.332837 16975 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae91a1e97a5f49a9a70bc3d90d12d07c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae91a1e97a5f49a9a70bc3d90d12d07c, State: Initialized, Role: FOLLOWER
I20260929 20:21:41.333410 16975 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae91a1e97a5f49a9a70bc3d90d12d07c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae91a1e97a5f49a9a70bc3d90d12d07c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41749 } }
I20260929 20:21:41.333639 16975 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae91a1e97a5f49a9a70bc3d90d12d07c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:21:41.333793 16975 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae91a1e97a5f49a9a70bc3d90d12d07c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:21:41.333962 16975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae91a1e97a5f49a9a70bc3d90d12d07c [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:41.336447 16975 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae91a1e97a5f49a9a70bc3d90d12d07c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae91a1e97a5f49a9a70bc3d90d12d07c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41749 } }
I20260929 20:21:41.336994 16975 leader_election.cc:302] T 00000000000000000000000000000000 P ae91a1e97a5f49a9a70bc3d90d12d07c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae91a1e97a5f49a9a70bc3d90d12d07c; no voters:
I20260929 20:21:41.337560 16975 leader_election.cc:288] T 00000000000000000000000000000000 P ae91a1e97a5f49a9a70bc3d90d12d07c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:21:41.337733 16985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae91a1e97a5f49a9a70bc3d90d12d07c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:21:41.338402 16985 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae91a1e97a5f49a9a70bc3d90d12d07c [term 1 LEADER]: Becoming Leader. State: Replica: ae91a1e97a5f49a9a70bc3d90d12d07c, State: Running, Role: LEADER
I20260929 20:21:41.338927 16985 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae91a1e97a5f49a9a70bc3d90d12d07c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae91a1e97a5f49a9a70bc3d90d12d07c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41749 } }
I20260929 20:21:41.339706 16975 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae91a1e97a5f49a9a70bc3d90d12d07c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:21:41.341833 16987 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae91a1e97a5f49a9a70bc3d90d12d07c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae91a1e97a5f49a9a70bc3d90d12d07c. Latest consensus state: current_term: 1 leader_uuid: "ae91a1e97a5f49a9a70bc3d90d12d07c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae91a1e97a5f49a9a70bc3d90d12d07c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41749 } } }
I20260929 20:21:41.341871 16986 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae91a1e97a5f49a9a70bc3d90d12d07c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae91a1e97a5f49a9a70bc3d90d12d07c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae91a1e97a5f49a9a70bc3d90d12d07c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41749 } } }
I20260929 20:21:41.342310 16987 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae91a1e97a5f49a9a70bc3d90d12d07c [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:41.342371 16986 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae91a1e97a5f49a9a70bc3d90d12d07c [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:41.344182 16990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:21:41.350211 16990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:21:41.359612 16990 catalog_manager.cc:1347] Generated new cluster ID: e13957b9058444fa9a1a62eb56a5d427
I20260929 20:21:41.359771 16990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:21:41.384483 16990 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:21:41.384977 16990 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:21:41.392009 16990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae91a1e97a5f49a9a70bc3d90d12d07c: Generated new TSK 0
I20260929 20:21:41.393033 16990 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:21:41.398537 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 16908
W20260929 20:21:41.399892 16970 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:32775
W20260929 20:21:41.400775 28389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 16908: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:21:41.400967 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 16908
2026-09-29T20:21:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1920 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:21:41 dist-test-slave-4cx3 krb5kdc[17006](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:21:41 dist-test-slave-4cx3 krb5kdc[17006](info): set up 2 sockets
Sep 29 20:21:41 dist-test-slave-4cx3 krb5kdc[17006](info): commencing operation
krb5kdc: starting...
W20260929 20:21:42.521366 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:21:42 dist-test-slave-4cx3 krb5kdc[17006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:21:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:21:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:21:42.787374 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33939
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:21:43.133278 17019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:43.133584 17019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:43.133744 17019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:43.141047 17019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:21:43.141162 17019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:43.141276 17019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:43.141394 17019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:21:43.141510 17019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:21:43.151597 17019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:43.152992 17019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:43.154348 17019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:43.166558 17024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:43.166528 17027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:43.166483 17025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:43.168749 17019 server_base.cc:1034] running on GCE node
I20260929 20:21:43.169555 17019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:43.170960 17019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:43.172163 17019 hybrid_clock.cc:648] HybridClock initialized: now 1790713303172118 us; error 33 us; skew 500 ppm
I20260929 20:21:43.172552 17019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:21:43 dist-test-slave-4cx3 krb5kdc[17006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:21:43.175829 17019 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:21:43.177387 17019 webserver.cc:460] Webserver started at http://127.0.0.1:36249/ using document root <none> and password file <none>
I20260929 20:21:43.178085 17019 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:43.178220 17019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:43.178550 17019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:43.181572 17019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "1d3b675ef92e49fb80c6a053a1d5d10d"
format_stamp: "Formatted at 2026-09-29 20:21:43 on dist-test-slave-4cx3"
server_key: "5793378e4c77550d7190c7efbf6c8f05"
server_key_iv: "1d2c08350e78b5a8731b314295591119"
server_key_version: "encryptionkey@0"
I20260929 20:21:43.182332 17019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "1d3b675ef92e49fb80c6a053a1d5d10d"
format_stamp: "Formatted at 2026-09-29 20:21:43 on dist-test-slave-4cx3"
server_key: "5793378e4c77550d7190c7efbf6c8f05"
server_key_iv: "1d2c08350e78b5a8731b314295591119"
server_key_version: "encryptionkey@0"
I20260929 20:21:43.187505 17019 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 20:21:43.190924 17034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:43.192334 17019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:21:43.192552 17019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "1d3b675ef92e49fb80c6a053a1d5d10d"
format_stamp: "Formatted at 2026-09-29 20:21:43 on dist-test-slave-4cx3"
server_key: "5793378e4c77550d7190c7efbf6c8f05"
server_key_iv: "1d2c08350e78b5a8731b314295591119"
server_key_version: "encryptionkey@0"
I20260929 20:21:43.192749 17019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:43.217128 17019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:43.221477 17019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:43.221779 17019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:43.240996 17019 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33939
I20260929 20:21:43.241020 17085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33939 every 8 connection(s)
I20260929 20:21:43.242455 17019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:21:43.243466 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 17019
I20260929 20:21:43.243654 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:21:43.244305 28389 external_mini_cluster.cc:1428] Setting key 7db91da4665d7f275bbaedc59546a52f
I20260929 20:21:43.247061 17086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 20:21:43 dist-test-slave-4cx3 krb5kdc[17006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 20:21:43.260526 17086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d3b675ef92e49fb80c6a053a1d5d10d: Bootstrap starting.
I20260929 20:21:43.264711 17086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d3b675ef92e49fb80c6a053a1d5d10d: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:43.265949 17086 log.cc:824] T 00000000000000000000000000000000 P 1d3b675ef92e49fb80c6a053a1d5d10d: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:43.269414 17086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d3b675ef92e49fb80c6a053a1d5d10d: No bootstrap required, opened a new log
I20260929 20:21:43.269935 17089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:43.246350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:59795 (local address 127.0.0.1:33939)
0929 20:21:43.247169 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:43.247184 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:43.247233 (+ 49us) server_negotiation.cc:407] Connection header received
0929 20:21:43.249642 (+ 2409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:43.249669 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:43.250202 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:43.250664 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:43.252157 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:43.254232 (+ 2075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:43.255568 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:43.256624 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:43.256817 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:43.261239 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:43.261249 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:43.261268 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:43.261318 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:43.265319 (+ 4001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:43.266131 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:43.266139 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:43.266146 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:43.266336 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:43.266858 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:43.266861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:43.266863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:43.267478 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:43.267795 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:43.268220 (+ 425us) server_negotiation.cc:299] Negotiation successful
0929 20:21:43.268566 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":319,"threads_started":1}
I20260929 20:21:43.274008 28389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:21:43.278165 17086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d3b675ef92e49fb80c6a053a1d5d10d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3b675ef92e49fb80c6a053a1d5d10d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33939 } }
I20260929 20:21:43.278726 17086 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d3b675ef92e49fb80c6a053a1d5d10d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:43.278870 17086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d3b675ef92e49fb80c6a053a1d5d10d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d3b675ef92e49fb80c6a053a1d5d10d, State: Initialized, Role: FOLLOWER
I20260929 20:21:43.279671 17086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d3b675ef92e49fb80c6a053a1d5d10d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3b675ef92e49fb80c6a053a1d5d10d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33939 } }
I20260929 20:21:43.279937 17086 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d3b675ef92e49fb80c6a053a1d5d10d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:21:43.280108 17086 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d3b675ef92e49fb80c6a053a1d5d10d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:21:43.280289 17086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d3b675ef92e49fb80c6a053a1d5d10d [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:43.282564 17086 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d3b675ef92e49fb80c6a053a1d5d10d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3b675ef92e49fb80c6a053a1d5d10d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33939 } }
I20260929 20:21:43.283102 17086 leader_election.cc:302] T 00000000000000000000000000000000 P 1d3b675ef92e49fb80c6a053a1d5d10d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d3b675ef92e49fb80c6a053a1d5d10d; no voters:
I20260929 20:21:43.283901 17086 leader_election.cc:288] T 00000000000000000000000000000000 P 1d3b675ef92e49fb80c6a053a1d5d10d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:21:43.284018 17096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d3b675ef92e49fb80c6a053a1d5d10d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:21:43.285218 17096 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d3b675ef92e49fb80c6a053a1d5d10d [term 1 LEADER]: Becoming Leader. State: Replica: 1d3b675ef92e49fb80c6a053a1d5d10d, State: Running, Role: LEADER
I20260929 20:21:43.285811 17096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d3b675ef92e49fb80c6a053a1d5d10d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3b675ef92e49fb80c6a053a1d5d10d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33939 } }
I20260929 20:21:43.286491 17086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d3b675ef92e49fb80c6a053a1d5d10d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:21:43.288934 17098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d3b675ef92e49fb80c6a053a1d5d10d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d3b675ef92e49fb80c6a053a1d5d10d. Latest consensus state: current_term: 1 leader_uuid: "1d3b675ef92e49fb80c6a053a1d5d10d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3b675ef92e49fb80c6a053a1d5d10d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33939 } } }
I20260929 20:21:43.289637 17098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d3b675ef92e49fb80c6a053a1d5d10d [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:43.289870 17097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d3b675ef92e49fb80c6a053a1d5d10d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d3b675ef92e49fb80c6a053a1d5d10d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3b675ef92e49fb80c6a053a1d5d10d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33939 } } }
I20260929 20:21:43.290181 17097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d3b675ef92e49fb80c6a053a1d5d10d [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:43.292577 17101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:21:43.298640 17101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:21:43.298996 17089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:43.278705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:44509 (local address 127.0.0.1:33939)
0929 20:21:43.278912 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:43.278920 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:43.279452 (+ 532us) server_negotiation.cc:407] Connection header received
0929 20:21:43.279676 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:43.279684 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:43.279756 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:43.279862 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:43.281478 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:43.282277 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:43.283726 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:43.284437 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:43.284597 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:43.287798 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:43.287805 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:43.287809 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:43.287846 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:43.292962 (+ 5116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:43.294193 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:43.294200 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:43.294203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:43.294532 (+ 329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:43.295287 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:43.295294 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:43.295300 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:43.295899 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:43.297748 (+ 1849us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:43.297988 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 20:21:43.298241 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 20:21:43.308915 17101 catalog_manager.cc:1347] Generated new cluster ID: 3e405b4559f0424aa26f9d0b764b2dcf
I20260929 20:21:43.309056 17101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:21:43.318423 17101 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:21:43.319971 17101 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:21:43.327314 17101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d3b675ef92e49fb80c6a053a1d5d10d: Generated new TSK 0
I20260929 20:21:43.328366 17101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:21:43.340161 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 17019
I20260929 20:21:43.341493 17081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:43.431605 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 17019
W20260929 20:21:43.449126 17004 connection.cc:535] client connection to 127.0.0.1:33939 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33939): Connection reset by peer (error 104)
2026-09-29T20:21:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2030 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:21:43 dist-test-slave-4cx3 krb5kdc[17118](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:21:43 dist-test-slave-4cx3 krb5kdc[17118](info): set up 2 sockets
Sep 29 20:21:43 dist-test-slave-4cx3 krb5kdc[17118](info): commencing operation
krb5kdc: starting...
W20260929 20:21:44.554762 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:21:44 dist-test-slave-4cx3 krb5kdc[17118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:21:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:21:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:21:44.825007 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37269
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:21:45.161084 17131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:45.161412 17131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:45.161599 17131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:45.168992 17131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:21:45.169112 17131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:45.169268 17131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:45.169404 17131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:21:45.169484 17131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:21:45.179805 17131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:45.181190 17131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:45.182576 17131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:45.195313 17136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:45.195358 17137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:45.195405 17139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:45.196187 17131 server_base.cc:1034] running on GCE node
I20260929 20:21:45.196978 17131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:45.198395 17131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:45.199621 17131 hybrid_clock.cc:648] HybridClock initialized: now 1790713305199582 us; error 32 us; skew 500 ppm
I20260929 20:21:45.199986 17131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:21:45 dist-test-slave-4cx3 krb5kdc[17118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:21:45.203161 17131 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:21:45.204784 17131 webserver.cc:460] Webserver started at http://127.0.0.1:46557/ using document root <none> and password file <none>
I20260929 20:21:45.205492 17131 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:45.205639 17131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:45.205962 17131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:45.209338 17131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "41c5187716ef4605912d9cfd00a663fb"
format_stamp: "Formatted at 2026-09-29 20:21:45 on dist-test-slave-4cx3"
server_key: "37c197c51b7a8c25c982763ac386079f"
server_key_iv: "a3a692584d4b54394e24c36420941a4d"
server_key_version: "encryptionkey@0"
I20260929 20:21:45.210135 17131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "41c5187716ef4605912d9cfd00a663fb"
format_stamp: "Formatted at 2026-09-29 20:21:45 on dist-test-slave-4cx3"
server_key: "37c197c51b7a8c25c982763ac386079f"
server_key_iv: "a3a692584d4b54394e24c36420941a4d"
server_key_version: "encryptionkey@0"
I20260929 20:21:45.215556 17131 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 20:21:45.219115 17146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:45.220827 17131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:21:45.221058 17131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "41c5187716ef4605912d9cfd00a663fb"
format_stamp: "Formatted at 2026-09-29 20:21:45 on dist-test-slave-4cx3"
server_key: "37c197c51b7a8c25c982763ac386079f"
server_key_iv: "a3a692584d4b54394e24c36420941a4d"
server_key_version: "encryptionkey@0"
I20260929 20:21:45.221283 17131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:45.238502 17131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:45.243541 17131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:45.243870 17131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:45.264317 17131 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37269
I20260929 20:21:45.264348 17197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37269 every 8 connection(s)
I20260929 20:21:45.265837 17131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:21:45.269874 17198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:45.270406 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 17131
I20260929 20:21:45.270594 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:21:45.271077 28389 external_mini_cluster.cc:1428] Setting key 1debbdef3150a60fe3a85c10e9ac2db5
I20260929 20:21:45.283942 17198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41c5187716ef4605912d9cfd00a663fb: Bootstrap starting.
I20260929 20:21:45.285496 17201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:45.273108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:52177 (local address 127.0.0.1:37269)
0929 20:21:45.273926 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:45.273940 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:45.273984 (+ 44us) server_negotiation.cc:407] Connection header received
0929 20:21:45.276068 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:45.276094 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:45.276559 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:45.276910 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:45.277560 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:45.278821 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:45.280806 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:45.281567 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:45.281702 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:45.283019 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:45.283027 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:45.283041 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:45.283682 (+ 641us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:45.283761 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:45.283786 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:45.283909 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:45.284292 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 20:21:45.284523 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":366,"threads_started":1}
I20260929 20:21:45.287314 17198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41c5187716ef4605912d9cfd00a663fb: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:45.288626 17198 log.cc:824] T 00000000000000000000000000000000 P 41c5187716ef4605912d9cfd00a663fb: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:45.289685 28389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:21:45.291718 17198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41c5187716ef4605912d9cfd00a663fb: No bootstrap required, opened a new log
I20260929 20:21:45.301695 17201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:45.294429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:49073 (local address 127.0.0.1:37269)
0929 20:21:45.294678 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:45.294685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:45.295046 (+ 361us) server_negotiation.cc:407] Connection header received
0929 20:21:45.295270 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:45.295274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:45.295357 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:45.295522 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:45.296178 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:45.296996 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:45.298904 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:45.299499 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:45.299608 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:45.300847 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:45.300851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:45.300854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:45.300871 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:45.300922 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:45.300941 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:45.301032 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:45.301243 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 20:21:45.301289 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 20:21:45.302141 17198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41c5187716ef4605912d9cfd00a663fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c5187716ef4605912d9cfd00a663fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37269 } }
I20260929 20:21:45.302717 17198 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41c5187716ef4605912d9cfd00a663fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:45.302830 17198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41c5187716ef4605912d9cfd00a663fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c5187716ef4605912d9cfd00a663fb, State: Initialized, Role: FOLLOWER
I20260929 20:21:45.303562 17198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41c5187716ef4605912d9cfd00a663fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c5187716ef4605912d9cfd00a663fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37269 } }
I20260929 20:21:45.303819 17198 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41c5187716ef4605912d9cfd00a663fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:21:45.303925 17198 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41c5187716ef4605912d9cfd00a663fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:21:45.304057 17198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41c5187716ef4605912d9cfd00a663fb [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:45.306607 17198 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41c5187716ef4605912d9cfd00a663fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c5187716ef4605912d9cfd00a663fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37269 } }
I20260929 20:21:45.307215 17198 leader_election.cc:302] T 00000000000000000000000000000000 P 41c5187716ef4605912d9cfd00a663fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41c5187716ef4605912d9cfd00a663fb; no voters:
I20260929 20:21:45.307824 17198 leader_election.cc:288] T 00000000000000000000000000000000 P 41c5187716ef4605912d9cfd00a663fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:21:45.307987 17208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41c5187716ef4605912d9cfd00a663fb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:21:45.308921 17208 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41c5187716ef4605912d9cfd00a663fb [term 1 LEADER]: Becoming Leader. State: Replica: 41c5187716ef4605912d9cfd00a663fb, State: Running, Role: LEADER
I20260929 20:21:45.309547 17208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41c5187716ef4605912d9cfd00a663fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c5187716ef4605912d9cfd00a663fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37269 } }
I20260929 20:21:45.310148 17198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41c5187716ef4605912d9cfd00a663fb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:21:45.312443 17209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41c5187716ef4605912d9cfd00a663fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41c5187716ef4605912d9cfd00a663fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c5187716ef4605912d9cfd00a663fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37269 } } }
I20260929 20:21:45.312923 17209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41c5187716ef4605912d9cfd00a663fb [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:45.315316 17210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41c5187716ef4605912d9cfd00a663fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41c5187716ef4605912d9cfd00a663fb. Latest consensus state: current_term: 1 leader_uuid: "41c5187716ef4605912d9cfd00a663fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c5187716ef4605912d9cfd00a663fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37269 } } }
I20260929 20:21:45.315651 17214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:21:45.315701 17210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41c5187716ef4605912d9cfd00a663fb [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:45.321158 17214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:21:45.330459 17214 catalog_manager.cc:1347] Generated new cluster ID: dc449b1876794894a5c336beef741acb
I20260929 20:21:45.330623 17214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:21:45.361567 17214 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:21:45.363620 17214 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:21:45.376730 17214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41c5187716ef4605912d9cfd00a663fb: Generated new TSK 0
I20260929 20:21:45.377910 17214 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:21:45.394358 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 17131
W20260929 20:21:45.395584 17193 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.220:52177
W20260929 20:21:45.396329 28389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 17131: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:21:45.396478 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 17131
W20260929 20:21:45.413443 17116 connection.cc:535] client connection to 127.0.0.1:37269 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37269): Connection reset by peer (error 104)
2026-09-29T20:21:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1964 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:21:45 dist-test-slave-4cx3 krb5kdc[17229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:21:45 dist-test-slave-4cx3 krb5kdc[17229](info): set up 2 sockets
Sep 29 20:21:45 dist-test-slave-4cx3 krb5kdc[17229](info): commencing operation
krb5kdc: starting...
W20260929 20:21:46.517855 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:21:46 dist-test-slave-4cx3 krb5kdc[17229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:21:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:21:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:21:46.795645 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37183
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:21:47.135059 17242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:47.135434 17242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:47.135584 17242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:47.143042 17242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:21:47.143167 17242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:47.143260 17242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:47.143380 17242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:21:47.143501 17242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:21:47.153908 17242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:47.155370 17242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:47.156803 17242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:47.169476 17248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:47.169562 17250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:47.169523 17247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:47.170701 17242 server_base.cc:1034] running on GCE node
I20260929 20:21:47.171700 17242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:47.173223 17242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:47.174470 17242 hybrid_clock.cc:648] HybridClock initialized: now 1790713307174417 us; error 35 us; skew 500 ppm
I20260929 20:21:47.174823 17242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:21:47 dist-test-slave-4cx3 krb5kdc[17229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:21:47.182898 17242 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:21:47.184675 17242 webserver.cc:460] Webserver started at http://127.0.0.1:43507/ using document root <none> and password file <none>
I20260929 20:21:47.185432 17242 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:47.185593 17242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:47.185915 17242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:47.189069 17242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "d4b846c4636149b0b502b8bb45b30bb7"
format_stamp: "Formatted at 2026-09-29 20:21:47 on dist-test-slave-4cx3"
server_key: "07afd81d848a94e8520b112911b94482"
server_key_iv: "d9a4054bf05e75e727a27d74e5454dfb"
server_key_version: "encryptionkey@0"
I20260929 20:21:47.189890 17242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "d4b846c4636149b0b502b8bb45b30bb7"
format_stamp: "Formatted at 2026-09-29 20:21:47 on dist-test-slave-4cx3"
server_key: "07afd81d848a94e8520b112911b94482"
server_key_iv: "d9a4054bf05e75e727a27d74e5454dfb"
server_key_version: "encryptionkey@0"
I20260929 20:21:47.195439 17242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 20:21:47.199139 17257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:47.200588 17242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 20:21:47.200786 17242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "d4b846c4636149b0b502b8bb45b30bb7"
format_stamp: "Formatted at 2026-09-29 20:21:47 on dist-test-slave-4cx3"
server_key: "07afd81d848a94e8520b112911b94482"
server_key_iv: "d9a4054bf05e75e727a27d74e5454dfb"
server_key_version: "encryptionkey@0"
I20260929 20:21:47.201020 17242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:47.206984 17242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:47.211184 17242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:47.211550 17242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:47.231021 17242 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37183
I20260929 20:21:47.231051 17308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37183 every 8 connection(s)
I20260929 20:21:47.232640 17242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:21:47.236521 17309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:47.241534 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 17242
I20260929 20:21:47.241742 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:21:47.242187 28389 external_mini_cluster.cc:1428] Setting key 2d85f237aea0bec278213b033b936ea8
I20260929 20:21:47.247654 17309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4b846c4636149b0b502b8bb45b30bb7: Bootstrap starting.
I20260929 20:21:47.250344 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:47.244038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:39109 (local address 127.0.0.1:37183)
0929 20:21:47.244852 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:47.244871 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:47.244917 (+ 46us) server_negotiation.cc:407] Connection header received
0929 20:21:47.246611 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:47.246633 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:47.246982 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:47.247300 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:47.248591 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:47.248598 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:47.248611 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:47.248626 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:47.248749 (+ 123us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:47.248798 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:47.249044 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:47.249150 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 20:21:47.249384 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":370,"threads_started":1}
I20260929 20:21:47.251438 17309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4b846c4636149b0b502b8bb45b30bb7: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:47.252548 17309 log.cc:824] T 00000000000000000000000000000000 P d4b846c4636149b0b502b8bb45b30bb7: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:47.254311 28389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:21:47.256119 17309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4b846c4636149b0b502b8bb45b30bb7: No bootstrap required, opened a new log
I20260929 20:21:47.262415 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:47.259779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:33701 (local address 127.0.0.1:37183)
0929 20:21:47.259948 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:47.259953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:47.260348 (+ 395us) server_negotiation.cc:407] Connection header received
0929 20:21:47.260501 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:47.260504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:47.260556 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:47.260640 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:47.261684 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:47.261690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:47.261696 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:47.261712 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:47.261755 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:47.261771 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:47.261876 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:47.262002 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 20:21:47.262057 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 20:21:47.265834 17309 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4b846c4636149b0b502b8bb45b30bb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b846c4636149b0b502b8bb45b30bb7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37183 } }
I20260929 20:21:47.266265 17309 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4b846c4636149b0b502b8bb45b30bb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:47.266369 17309 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4b846c4636149b0b502b8bb45b30bb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4b846c4636149b0b502b8bb45b30bb7, State: Initialized, Role: FOLLOWER
I20260929 20:21:47.266959 17309 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4b846c4636149b0b502b8bb45b30bb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b846c4636149b0b502b8bb45b30bb7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37183 } }
I20260929 20:21:47.267234 17309 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4b846c4636149b0b502b8bb45b30bb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:21:47.267386 17309 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4b846c4636149b0b502b8bb45b30bb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:21:47.267606 17309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4b846c4636149b0b502b8bb45b30bb7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:47.269949 17309 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4b846c4636149b0b502b8bb45b30bb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b846c4636149b0b502b8bb45b30bb7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37183 } }
I20260929 20:21:47.270480 17309 leader_election.cc:302] T 00000000000000000000000000000000 P d4b846c4636149b0b502b8bb45b30bb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4b846c4636149b0b502b8bb45b30bb7; no voters:
I20260929 20:21:47.271095 17309 leader_election.cc:288] T 00000000000000000000000000000000 P d4b846c4636149b0b502b8bb45b30bb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:21:47.271245 17319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4b846c4636149b0b502b8bb45b30bb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:21:47.272392 17319 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4b846c4636149b0b502b8bb45b30bb7 [term 1 LEADER]: Becoming Leader. State: Replica: d4b846c4636149b0b502b8bb45b30bb7, State: Running, Role: LEADER
I20260929 20:21:47.273003 17319 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4b846c4636149b0b502b8bb45b30bb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b846c4636149b0b502b8bb45b30bb7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37183 } }
I20260929 20:21:47.273501 17309 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4b846c4636149b0b502b8bb45b30bb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:21:47.275997 17321 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b846c4636149b0b502b8bb45b30bb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4b846c4636149b0b502b8bb45b30bb7. Latest consensus state: current_term: 1 leader_uuid: "d4b846c4636149b0b502b8bb45b30bb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b846c4636149b0b502b8bb45b30bb7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37183 } } }
I20260929 20:21:47.276254 17320 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b846c4636149b0b502b8bb45b30bb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4b846c4636149b0b502b8bb45b30bb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b846c4636149b0b502b8bb45b30bb7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37183 } } }
I20260929 20:21:47.276541 17321 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b846c4636149b0b502b8bb45b30bb7 [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:47.276571 17320 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b846c4636149b0b502b8bb45b30bb7 [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:47.279155 17328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:21:47.284614 17328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:21:47.294169 17328 catalog_manager.cc:1347] Generated new cluster ID: 8df50db0510e4a1fba930c1b3af59a4d
I20260929 20:21:47.294355 17328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:21:47.304646 17328 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:21:47.305078 17328 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:21:47.317785 17328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4b846c4636149b0b502b8bb45b30bb7: Generated new TSK 0
I20260929 20:21:47.318953 17328 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:21:47.328994 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 17242
W20260929 20:21:47.330245 17304 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.220:39109
W20260929 20:21:47.331059 28389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 17242: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:21:47.331218 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 17242
2026-09-29T20:21:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1934 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:21:47 dist-test-slave-4cx3 krb5kdc[17340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:21:47 dist-test-slave-4cx3 krb5kdc[17340](info): set up 2 sockets
Sep 29 20:21:47 dist-test-slave-4cx3 krb5kdc[17340](info): commencing operation
krb5kdc: starting...
W20260929 20:21:48.466791 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:21:48 dist-test-slave-4cx3 krb5kdc[17340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:21:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:21:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:21:48.739529 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36777
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:21:49.075599 17353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:49.075909 17353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:49.076013 17353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:49.083280 17353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:21:49.083420 17353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:49.083469 17353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:49.083525 17353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:21:49.083570 17353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:21:49.093565 17353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36777
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:49.095005 17353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:49.096321 17353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:49.108392 17361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:49.108381 17359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:49.108758 17358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:49.109347 17353 server_base.cc:1034] running on GCE node
I20260929 20:21:49.110239 17353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:49.111796 17353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:49.113029 17353 hybrid_clock.cc:648] HybridClock initialized: now 1790713309112992 us; error 47 us; skew 500 ppm
I20260929 20:21:49.113415 17353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:21:49 dist-test-slave-4cx3 krb5kdc[17340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:21:49.116806 17353 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:21:49.118456 17353 webserver.cc:460] Webserver started at http://127.0.0.1:36277/ using document root <none> and password file <none>
I20260929 20:21:49.119189 17353 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:49.119342 17353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:49.119690 17353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:49.122699 17353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "893ddc151d314cbeb59303299acfb56a"
format_stamp: "Formatted at 2026-09-29 20:21:49 on dist-test-slave-4cx3"
server_key: "88439a90d0f4a987e642c06d131934b0"
server_key_iv: "b7c902ccfdbb391167b80a72389641c5"
server_key_version: "encryptionkey@0"
I20260929 20:21:49.123575 17353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "893ddc151d314cbeb59303299acfb56a"
format_stamp: "Formatted at 2026-09-29 20:21:49 on dist-test-slave-4cx3"
server_key: "88439a90d0f4a987e642c06d131934b0"
server_key_iv: "b7c902ccfdbb391167b80a72389641c5"
server_key_version: "encryptionkey@0"
I20260929 20:21:49.128722 17353 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 20:21:49.132365 17368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:49.133738 17353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:21:49.133944 17353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "893ddc151d314cbeb59303299acfb56a"
format_stamp: "Formatted at 2026-09-29 20:21:49 on dist-test-slave-4cx3"
server_key: "88439a90d0f4a987e642c06d131934b0"
server_key_iv: "b7c902ccfdbb391167b80a72389641c5"
server_key_version: "encryptionkey@0"
I20260929 20:21:49.134156 17353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:49.157073 17353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:49.161758 17353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:49.162076 17353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:49.181149 17353 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36777
I20260929 20:21:49.181193 17419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36777 every 8 connection(s)
I20260929 20:21:49.182555 17353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:21:49.185894 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 17353
I20260929 20:21:49.186097 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:21:49.186566 28389 external_mini_cluster.cc:1428] Setting key a269b0bafade83adcc68ea4739331e9a
I20260929 20:21:49.186607 17420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:49.201385 17420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 893ddc151d314cbeb59303299acfb56a: Bootstrap starting.
Sep 29 20:21:49 dist-test-slave-4cx3 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 20:21:49.205255 17420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 893ddc151d314cbeb59303299acfb56a: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:49.206691 17420 log.cc:824] T 00000000000000000000000000000000 P 893ddc151d314cbeb59303299acfb56a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:49.210453 17420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 893ddc151d314cbeb59303299acfb56a: No bootstrap required, opened a new log
I20260929 20:21:49.212328 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:49.188561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:45973 (local address 127.0.0.1:36777)
0929 20:21:49.189407 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:49.189423 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:49.189472 (+ 49us) server_negotiation.cc:407] Connection header received
0929 20:21:49.191822 (+ 2350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:49.191855 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:49.192480 (+ 625us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:49.192977 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:49.194395 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:49.196389 (+ 1994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:49.197669 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:49.198796 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:49.198989 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:49.203237 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:49.203248 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:49.203267 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:49.203320 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:49.207178 (+ 3858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:49.208386 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:49.208394 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:49.208402 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:49.208606 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:49.209117 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:49.209122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:49.209124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:49.209828 (+ 704us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:49.210148 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:49.210598 (+ 450us) server_negotiation.cc:299] Negotiation successful
0929 20:21:49.210951 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":335,"threads_started":1}
I20260929 20:21:49.216620 28389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:21:49.219784 17420 raft_consensus.cc:348] T 00000000000000000000000000000000 P 893ddc151d314cbeb59303299acfb56a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893ddc151d314cbeb59303299acfb56a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36777 } }
I20260929 20:21:49.220191 17420 raft_consensus.cc:374] T 00000000000000000000000000000000 P 893ddc151d314cbeb59303299acfb56a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:49.220315 17420 raft_consensus.cc:729] T 00000000000000000000000000000000 P 893ddc151d314cbeb59303299acfb56a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 893ddc151d314cbeb59303299acfb56a, State: Initialized, Role: FOLLOWER
I20260929 20:21:49.220963 17420 consensus_queue.cc:260] T 00000000000000000000000000000000 P 893ddc151d314cbeb59303299acfb56a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893ddc151d314cbeb59303299acfb56a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36777 } }
I20260929 20:21:49.221242 17420 raft_consensus.cc:388] T 00000000000000000000000000000000 P 893ddc151d314cbeb59303299acfb56a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:21:49.221405 17420 raft_consensus.cc:482] T 00000000000000000000000000000000 P 893ddc151d314cbeb59303299acfb56a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:21:49.221659 17420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 893ddc151d314cbeb59303299acfb56a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:49.224995 17420 raft_consensus.cc:504] T 00000000000000000000000000000000 P 893ddc151d314cbeb59303299acfb56a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893ddc151d314cbeb59303299acfb56a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36777 } }
I20260929 20:21:49.225667 17420 leader_election.cc:302] T 00000000000000000000000000000000 P 893ddc151d314cbeb59303299acfb56a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 893ddc151d314cbeb59303299acfb56a; no voters:
I20260929 20:21:49.226343 17420 leader_election.cc:288] T 00000000000000000000000000000000 P 893ddc151d314cbeb59303299acfb56a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:21:49.226480 17430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 893ddc151d314cbeb59303299acfb56a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:21:49.227769 17430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 893ddc151d314cbeb59303299acfb56a [term 1 LEADER]: Becoming Leader. State: Replica: 893ddc151d314cbeb59303299acfb56a, State: Running, Role: LEADER
I20260929 20:21:49.228453 17430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 893ddc151d314cbeb59303299acfb56a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893ddc151d314cbeb59303299acfb56a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36777 } }
I20260929 20:21:49.228761 17420 sys_catalog.cc:564] T 00000000000000000000000000000000 P 893ddc151d314cbeb59303299acfb56a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:21:49.236292 17432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 893ddc151d314cbeb59303299acfb56a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 893ddc151d314cbeb59303299acfb56a. Latest consensus state: current_term: 1 leader_uuid: "893ddc151d314cbeb59303299acfb56a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893ddc151d314cbeb59303299acfb56a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36777 } } }
I20260929 20:21:49.236821 17432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 893ddc151d314cbeb59303299acfb56a [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:49.237550 17431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 893ddc151d314cbeb59303299acfb56a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "893ddc151d314cbeb59303299acfb56a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893ddc151d314cbeb59303299acfb56a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36777 } } }
I20260929 20:21:49.237885 17431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 893ddc151d314cbeb59303299acfb56a [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:49.241876 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:49.221407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:60145 (local address 127.0.0.1:36777)
0929 20:21:49.221645 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:49.221655 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:49.222136 (+ 481us) server_negotiation.cc:407] Connection header received
0929 20:21:49.222322 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:49.222326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:49.222395 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:49.222499 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:49.224413 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:49.225218 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:49.227139 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:49.227827 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:49.227944 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:49.232180 (+ 4236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:49.232191 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:49.232199 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:49.232260 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:49.236892 (+ 4632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:49.237925 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:49.237943 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:49.237950 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:49.238093 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:49.238784 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:49.238793 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:49.238799 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:49.239347 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:49.239715 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:49.241111 (+ 1396us) server_negotiation.cc:299] Negotiation successful
0929 20:21:49.241360 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:21:49.243892 17444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:21:49.248009 17444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:21:49.256763 17444 catalog_manager.cc:1347] Generated new cluster ID: 501ee46cdc02461fabc04ec9af38edf3
I20260929 20:21:49.256909 17444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:21:49.273658 17444 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:21:49.274987 17444 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:21:49.284792 17444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 893ddc151d314cbeb59303299acfb56a: Generated new TSK 0
I20260929 20:21:49.285740 17444 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:21:49.295492 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 17353
I20260929 20:21:49.296707 17415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:49.388929 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 17353
W20260929 20:21:49.406144 17338 connection.cc:535] client connection to 127.0.0.1:36777 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36777): Connection reset by peer (error 104)
2026-09-29T20:21:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2057 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:21:49 dist-test-slave-4cx3 krb5kdc[17452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:21:49 dist-test-slave-4cx3 krb5kdc[17452](info): set up 2 sockets
Sep 29 20:21:49 dist-test-slave-4cx3 krb5kdc[17452](info): commencing operation
krb5kdc: starting...
W20260929 20:21:50.502840 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:21:50 dist-test-slave-4cx3 krb5kdc[17452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:21:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:21:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:21:50.770774 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32937
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:21:51.104096 17465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:51.104390 17465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:51.104562 17465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:51.111877 17465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:21:51.112000 17465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:51.112136 17465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:51.112239 17465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:21:51.112310 17465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:21:51.122385 17465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32937
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:51.123996 17465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:51.125391 17465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:51.137742 17470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:51.137720 17473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:51.137688 17471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:51.138542 17465 server_base.cc:1034] running on GCE node
I20260929 20:21:51.139290 17465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:51.140841 17465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:51.142133 17465 hybrid_clock.cc:648] HybridClock initialized: now 1790713311142089 us; error 37 us; skew 500 ppm
I20260929 20:21:51.142479 17465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:21:51 dist-test-slave-4cx3 krb5kdc[17452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:21:51.145695 17465 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:21:51.147281 17465 webserver.cc:460] Webserver started at http://127.0.0.1:36423/ using document root <none> and password file <none>
I20260929 20:21:51.148026 17465 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:51.148134 17465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:51.148384 17465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:51.151371 17465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "f92b712de861494b866f17e6ca3462cf"
format_stamp: "Formatted at 2026-09-29 20:21:51 on dist-test-slave-4cx3"
server_key: "82eaf0090b65610240ac0ed603774e4a"
server_key_iv: "19e2e26cfca539f0a7c06349ad4caaf3"
server_key_version: "encryptionkey@0"
I20260929 20:21:51.152184 17465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "f92b712de861494b866f17e6ca3462cf"
format_stamp: "Formatted at 2026-09-29 20:21:51 on dist-test-slave-4cx3"
server_key: "82eaf0090b65610240ac0ed603774e4a"
server_key_iv: "19e2e26cfca539f0a7c06349ad4caaf3"
server_key_version: "encryptionkey@0"
I20260929 20:21:51.157270 17465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:21:51.160818 17480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:51.162284 17465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:21:51.162513 17465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "f92b712de861494b866f17e6ca3462cf"
format_stamp: "Formatted at 2026-09-29 20:21:51 on dist-test-slave-4cx3"
server_key: "82eaf0090b65610240ac0ed603774e4a"
server_key_iv: "19e2e26cfca539f0a7c06349ad4caaf3"
server_key_version: "encryptionkey@0"
I20260929 20:21:51.162719 17465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:51.184542 17465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:51.188791 17465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:51.189092 17465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:51.207983 17465 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32937
I20260929 20:21:51.208029 17531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32937 every 8 connection(s)
I20260929 20:21:51.209412 17465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:21:51.213203 17532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:51.215658 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 17465
I20260929 20:21:51.215837 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:21:51.216315 28389 external_mini_cluster.cc:1428] Setting key a8c0da23214f4b286a8624fc295d6460
I20260929 20:21:51.225145 17532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f92b712de861494b866f17e6ca3462cf: Bootstrap starting.
I20260929 20:21:51.228538 17532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f92b712de861494b866f17e6ca3462cf: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:51.229669 17532 log.cc:824] T 00000000000000000000000000000000 P f92b712de861494b866f17e6ca3462cf: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:51.229945 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:51.218124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:44345 (local address 127.0.0.1:32937)
0929 20:21:51.218996 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:51.219009 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:51.219055 (+ 46us) server_negotiation.cc:407] Connection header received
0929 20:21:51.220820 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:51.220845 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:51.221281 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:51.221762 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:51.222354 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:51.223803 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:51.225228 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:51.225962 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:51.226161 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:51.227541 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:51.227549 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:51.227562 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:51.228094 (+ 532us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:51.228171 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:51.228197 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:51.228367 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:51.228643 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 20:21:51.228832 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":384,"threads_started":1}
I20260929 20:21:51.232839 17532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f92b712de861494b866f17e6ca3462cf: No bootstrap required, opened a new log
I20260929 20:21:51.233760 28389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:21:51.243108 17532 raft_consensus.cc:348] T 00000000000000000000000000000000 P f92b712de861494b866f17e6ca3462cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f92b712de861494b866f17e6ca3462cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32937 } }
I20260929 20:21:51.243626 17532 raft_consensus.cc:374] T 00000000000000000000000000000000 P f92b712de861494b866f17e6ca3462cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:51.243738 17532 raft_consensus.cc:729] T 00000000000000000000000000000000 P f92b712de861494b866f17e6ca3462cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f92b712de861494b866f17e6ca3462cf, State: Initialized, Role: FOLLOWER
I20260929 20:21:51.244371 17532 consensus_queue.cc:260] T 00000000000000000000000000000000 P f92b712de861494b866f17e6ca3462cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f92b712de861494b866f17e6ca3462cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32937 } }
I20260929 20:21:51.244668 17532 raft_consensus.cc:388] T 00000000000000000000000000000000 P f92b712de861494b866f17e6ca3462cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:21:51.244834 17532 raft_consensus.cc:482] T 00000000000000000000000000000000 P f92b712de861494b866f17e6ca3462cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:21:51.245020 17532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f92b712de861494b866f17e6ca3462cf [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:51.246068 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:51.239108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:39839 (local address 127.0.0.1:32937)
0929 20:21:51.239307 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:51.239317 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:51.239740 (+ 423us) server_negotiation.cc:407] Connection header received
0929 20:21:51.239894 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:51.239899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:51.240005 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:51.240122 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:51.240669 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:51.241852 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:51.243122 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:51.243824 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:51.243961 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:51.245218 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:51.245226 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:51.245231 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:51.245251 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:51.245300 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:51.245326 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:51.245468 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:51.245654 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 20:21:51.245717 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:21:51.247835 17532 raft_consensus.cc:504] T 00000000000000000000000000000000 P f92b712de861494b866f17e6ca3462cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f92b712de861494b866f17e6ca3462cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32937 } }
I20260929 20:21:51.248271 17532 leader_election.cc:302] T 00000000000000000000000000000000 P f92b712de861494b866f17e6ca3462cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f92b712de861494b866f17e6ca3462cf; no voters:
I20260929 20:21:51.248805 17532 leader_election.cc:288] T 00000000000000000000000000000000 P f92b712de861494b866f17e6ca3462cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:21:51.248940 17542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f92b712de861494b866f17e6ca3462cf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:21:51.249727 17542 raft_consensus.cc:686] T 00000000000000000000000000000000 P f92b712de861494b866f17e6ca3462cf [term 1 LEADER]: Becoming Leader. State: Replica: f92b712de861494b866f17e6ca3462cf, State: Running, Role: LEADER
I20260929 20:21:51.250309 17542 consensus_queue.cc:237] T 00000000000000000000000000000000 P f92b712de861494b866f17e6ca3462cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f92b712de861494b866f17e6ca3462cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32937 } }
I20260929 20:21:51.251024 17532 sys_catalog.cc:564] T 00000000000000000000000000000000 P f92b712de861494b866f17e6ca3462cf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:21:51.254621 17544 sys_catalog.cc:455] T 00000000000000000000000000000000 P f92b712de861494b866f17e6ca3462cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader f92b712de861494b866f17e6ca3462cf. Latest consensus state: current_term: 1 leader_uuid: "f92b712de861494b866f17e6ca3462cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f92b712de861494b866f17e6ca3462cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32937 } } }
I20260929 20:21:51.255086 17544 sys_catalog.cc:458] T 00000000000000000000000000000000 P f92b712de861494b866f17e6ca3462cf [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:51.255724 17543 sys_catalog.cc:455] T 00000000000000000000000000000000 P f92b712de861494b866f17e6ca3462cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f92b712de861494b866f17e6ca3462cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f92b712de861494b866f17e6ca3462cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32937 } } }
I20260929 20:21:51.256088 17543 sys_catalog.cc:458] T 00000000000000000000000000000000 P f92b712de861494b866f17e6ca3462cf [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:51.261065 17552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:21:51.268151 17552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:21:51.278491 17552 catalog_manager.cc:1347] Generated new cluster ID: 4a5a0dbf2f0f462ca39ed1eefc8185a4
I20260929 20:21:51.278630 17552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:21:51.293583 17552 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:21:51.294883 17552 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:21:51.306200 17552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f92b712de861494b866f17e6ca3462cf: Generated new TSK 0
I20260929 20:21:51.307089 17552 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:21:51.320681 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 17465
W20260929 20:21:51.321949 17527 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.220:44345
W20260929 20:21:51.322701 28389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 17465: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:21:51.322885 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 17465
W20260929 20:21:51.337280 17450 connection.cc:535] client connection to 127.0.0.1:32937 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32937): Connection reset by peer (error 104)
2026-09-29T20:21:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1930 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:21:51 dist-test-slave-4cx3 krb5kdc[17563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:21:51 dist-test-slave-4cx3 krb5kdc[17563](info): set up 2 sockets
Sep 29 20:21:51 dist-test-slave-4cx3 krb5kdc[17563](info): commencing operation
krb5kdc: starting...
W20260929 20:21:52.450910 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:21:52 dist-test-slave-4cx3 krb5kdc[17563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:21:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:21:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:21:52.730760 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40967
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:21:53.082554 17576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:53.082871 17576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:53.082966 17576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:53.090423 17576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:21:53.090543 17576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:53.090595 17576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:53.090663 17576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:21:53.090709 17576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:21:53.101001 17576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40967
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790712966917858-28389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:53.102281 17576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:53.103603 17576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:53.115487 17582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:53.115561 17584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:53.115561 17581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:53.116736 17576 server_base.cc:1034] running on GCE node
I20260929 20:21:53.117588 17576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:53.119081 17576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:53.120307 17576 hybrid_clock.cc:648] HybridClock initialized: now 1790713313120205 us; error 96 us; skew 500 ppm
I20260929 20:21:53.120728 17576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:21:53 dist-test-slave-4cx3 krb5kdc[17563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:21:53.123847 17576 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:21:53.125253 17576 webserver.cc:460] Webserver started at http://127.0.0.1:33433/ using document root <none> and password file <none>
I20260929 20:21:53.126025 17576 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:53.126242 17576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:53.126577 17576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:53.129673 17576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "57a46407d3d9495cacaae574f268d45a"
format_stamp: "Formatted at 2026-09-29 20:21:53 on dist-test-slave-4cx3"
server_key: "ec71236bf54ca28dafad5c443975c1c7"
server_key_iv: "628ed59d5fa7505a1078afddb3706513"
server_key_version: "encryptionkey@0"
I20260929 20:21:53.130452 17576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "57a46407d3d9495cacaae574f268d45a"
format_stamp: "Formatted at 2026-09-29 20:21:53 on dist-test-slave-4cx3"
server_key: "ec71236bf54ca28dafad5c443975c1c7"
server_key_iv: "628ed59d5fa7505a1078afddb3706513"
server_key_version: "encryptionkey@0"
I20260929 20:21:53.135736 17576 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:21:53.139178 17591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:53.140658 17576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:21:53.140841 17576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "57a46407d3d9495cacaae574f268d45a"
format_stamp: "Formatted at 2026-09-29 20:21:53 on dist-test-slave-4cx3"
server_key: "ec71236bf54ca28dafad5c443975c1c7"
server_key_iv: "628ed59d5fa7505a1078afddb3706513"
server_key_version: "encryptionkey@0"
I20260929 20:21:53.141042 17576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:53.146888 17576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:53.150781 17576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:53.151082 17576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:53.169989 17576 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40967
I20260929 20:21:53.170030 17642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40967 every 8 connection(s)
I20260929 20:21:53.171454 17576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:21:53.175175 17643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:53.178185 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 17576
I20260929 20:21:53.178426 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:21:53.178885 28389 external_mini_cluster.cc:1428] Setting key c65b0941df6688a78587766e135febed
I20260929 20:21:53.188058 17646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:53.180888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:34627 (local address 127.0.0.1:40967)
0929 20:21:53.181679 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:53.181695 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:53.181744 (+ 49us) server_negotiation.cc:407] Connection header received
0929 20:21:53.184126 (+ 2382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:53.184161 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:53.184658 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:53.185117 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:53.186131 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:53.186139 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:53.186156 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:53.186178 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:53.186252 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:53.186278 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:53.186520 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:53.186664 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 20:21:53.186851 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":299,"threads_started":1}
I20260929 20:21:53.188799 17643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57a46407d3d9495cacaae574f268d45a: Bootstrap starting.
I20260929 20:21:53.191741 28389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:21:53.192214 17643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57a46407d3d9495cacaae574f268d45a: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:53.193254 17643 log.cc:824] T 00000000000000000000000000000000 P 57a46407d3d9495cacaae574f268d45a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:53.196357 17643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57a46407d3d9495cacaae574f268d45a: No bootstrap required, opened a new log
I20260929 20:21:53.198850 17646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:53.196295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:56019 (local address 127.0.0.1:40967)
0929 20:21:53.196485 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:53.196491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:53.196826 (+ 335us) server_negotiation.cc:407] Connection header received
0929 20:21:53.196962 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:53.196967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:53.197022 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:53.197137 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:53.198133 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:53.198139 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:53.198144 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:53.198162 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:53.198210 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:53.198234 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:53.198365 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:53.198494 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 20:21:53.198558 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:21:53.205029 17643 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57a46407d3d9495cacaae574f268d45a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a46407d3d9495cacaae574f268d45a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40967 } }
I20260929 20:21:53.205420 17643 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57a46407d3d9495cacaae574f268d45a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:53.205525 17643 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57a46407d3d9495cacaae574f268d45a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57a46407d3d9495cacaae574f268d45a, State: Initialized, Role: FOLLOWER
I20260929 20:21:53.206208 17643 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57a46407d3d9495cacaae574f268d45a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a46407d3d9495cacaae574f268d45a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40967 } }
I20260929 20:21:53.206425 17643 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57a46407d3d9495cacaae574f268d45a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:21:53.206593 17643 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57a46407d3d9495cacaae574f268d45a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:21:53.206758 17643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57a46407d3d9495cacaae574f268d45a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:53.209388 17643 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57a46407d3d9495cacaae574f268d45a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a46407d3d9495cacaae574f268d45a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40967 } }
I20260929 20:21:53.209918 17643 leader_election.cc:302] T 00000000000000000000000000000000 P 57a46407d3d9495cacaae574f268d45a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57a46407d3d9495cacaae574f268d45a; no voters:
I20260929 20:21:53.210518 17643 leader_election.cc:288] T 00000000000000000000000000000000 P 57a46407d3d9495cacaae574f268d45a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:21:53.210685 17653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57a46407d3d9495cacaae574f268d45a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:21:53.211377 17653 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57a46407d3d9495cacaae574f268d45a [term 1 LEADER]: Becoming Leader. State: Replica: 57a46407d3d9495cacaae574f268d45a, State: Running, Role: LEADER
I20260929 20:21:53.211961 17653 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57a46407d3d9495cacaae574f268d45a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a46407d3d9495cacaae574f268d45a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40967 } }
I20260929 20:21:53.212699 17643 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57a46407d3d9495cacaae574f268d45a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:21:53.215049 17654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57a46407d3d9495cacaae574f268d45a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57a46407d3d9495cacaae574f268d45a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a46407d3d9495cacaae574f268d45a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40967 } } }
I20260929 20:21:53.215173 17655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57a46407d3d9495cacaae574f268d45a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57a46407d3d9495cacaae574f268d45a. Latest consensus state: current_term: 1 leader_uuid: "57a46407d3d9495cacaae574f268d45a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a46407d3d9495cacaae574f268d45a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40967 } } }
I20260929 20:21:53.215513 17654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57a46407d3d9495cacaae574f268d45a [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:53.215601 17655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57a46407d3d9495cacaae574f268d45a [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:53.217645 17658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:21:53.222952 17658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:21:53.232510 17658 catalog_manager.cc:1347] Generated new cluster ID: 58317de6c87845899efc9ff4983944a9
I20260929 20:21:53.232636 17658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:21:53.275115 17658 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:21:53.275682 17658 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:21:53.292237 17658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57a46407d3d9495cacaae574f268d45a: Generated new TSK 0
I20260929 20:21:53.293334 17658 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:21:53.311501 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 17576
W20260929 20:21:53.313097 17638 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.220:34627
W20260929 20:21:53.313913 28389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 17576: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:21:53.314077 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 17576
2026-09-29T20:21:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1994 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:21:53 dist-test-slave-4cx3 krb5kdc[17674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:21:53 dist-test-slave-4cx3 krb5kdc[17674](info): set up 2 sockets
Sep 29 20:21:53 dist-test-slave-4cx3 krb5kdc[17674](info): commencing operation
krb5kdc: starting...
W20260929 20:21:54.438748 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:21:54 dist-test-slave-4cx3 krb5kdc[17674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:21:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:21:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
I20260929 20:21:54.698984 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:35353
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:45735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:35353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:21:55.046193 17687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:55.046577 17687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:55.046739 17687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:55.054569 17687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:21:55.054718 17687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:55.054826 17687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:55.054940 17687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:21:55.055029 17687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:21:55.066078 17687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:45735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:35353
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:35353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:55.067565 17687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:55.068946 17687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:55.082243 17692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:55.082270 17693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:55.082226 17695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:55.083335 17687 server_base.cc:1034] running on GCE node
I20260929 20:21:55.084236 17687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:55.085777 17687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:55.086985 17687 hybrid_clock.cc:648] HybridClock initialized: now 1790713315086929 us; error 35 us; skew 500 ppm
I20260929 20:21:55.087369 17687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:21:55 dist-test-slave-4cx3 krb5kdc[17674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713315, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:21:55.090776 17687 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:21:55.092373 17687 webserver.cc:460] Webserver started at http://127.27.185.126:42131/ using document root <none> and password file <none>
I20260929 20:21:55.093093 17687 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:55.093248 17687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:55.093577 17687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:55.096774 17687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "e1e4e725d37c4ee3bac78c6bfab74408"
format_stamp: "Formatted at 2026-09-29 20:21:55 on dist-test-slave-4cx3"
server_key: "6a5aad9f97590ce0c4338bc1e66e042c"
server_key_iv: "bad936dcb43b0d235b855a8232839bb0"
server_key_version: "encryptionkey@0"
I20260929 20:21:55.097617 17687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "e1e4e725d37c4ee3bac78c6bfab74408"
format_stamp: "Formatted at 2026-09-29 20:21:55 on dist-test-slave-4cx3"
server_key: "6a5aad9f97590ce0c4338bc1e66e042c"
server_key_iv: "bad936dcb43b0d235b855a8232839bb0"
server_key_version: "encryptionkey@0"
I20260929 20:21:55.103049 17687 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 20:21:55.106616 17702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:55.108105 17687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:21:55.108297 17687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "e1e4e725d37c4ee3bac78c6bfab74408"
format_stamp: "Formatted at 2026-09-29 20:21:55 on dist-test-slave-4cx3"
server_key: "6a5aad9f97590ce0c4338bc1e66e042c"
server_key_iv: "bad936dcb43b0d235b855a8232839bb0"
server_key_version: "encryptionkey@0"
I20260929 20:21:55.108511 17687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:55.136190 17687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:55.140878 17687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:55.141201 17687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:55.160498 17687 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:35353
I20260929 20:21:55.160506 17753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:35353 every 8 connection(s)
I20260929 20:21:55.161972 17687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:21:55.166007 17754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:55.166664 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 17687
I20260929 20:21:55.166849 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:21:55.167305 28389 external_mini_cluster.cc:1428] Setting key 407087b5bd7326caee19a1ebcc442e06
I20260929 20:21:55.180482 17754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1e4e725d37c4ee3bac78c6bfab74408: Bootstrap starting.
Sep 29 20:21:55 dist-test-slave-4cx3 krb5kdc[17674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:21:55.184239 17754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1e4e725d37c4ee3bac78c6bfab74408: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:55.185339 17754 log.cc:824] T 00000000000000000000000000000000 P e1e4e725d37c4ee3bac78c6bfab74408: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:55.188637 17754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1e4e725d37c4ee3bac78c6bfab74408: No bootstrap required, opened a new log
I20260929 20:21:55.192490 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:55.169186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60661 (local address 127.27.185.126:35353)
0929 20:21:55.170046 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:55.170060 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:55.170102 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:21:55.172227 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:55.172252 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:55.172694 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:55.173086 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:55.174950 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:55.176447 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:55.178309 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:55.179006 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:55.179136 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:55.184071 (+ 4935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:55.184079 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:55.184092 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:55.184134 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:55.187826 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:55.188604 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:55.188615 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:55.188630 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:55.188844 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:55.189282 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:55.189287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:55.189290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:55.189981 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:55.190303 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:55.190711 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 20:21:55.191074 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":420,"threads_started":1}
I20260929 20:21:55.196599 28389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:21:55.198320 17754 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1e4e725d37c4ee3bac78c6bfab74408 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e4e725d37c4ee3bac78c6bfab74408" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 35353 } }
I20260929 20:21:55.198696 17754 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1e4e725d37c4ee3bac78c6bfab74408 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:55.198808 17754 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1e4e725d37c4ee3bac78c6bfab74408 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1e4e725d37c4ee3bac78c6bfab74408, State: Initialized, Role: FOLLOWER
I20260929 20:21:55.199508 17754 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1e4e725d37c4ee3bac78c6bfab74408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e4e725d37c4ee3bac78c6bfab74408" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 35353 } }
I20260929 20:21:55.199764 17754 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1e4e725d37c4ee3bac78c6bfab74408 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:21:55.199931 17754 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1e4e725d37c4ee3bac78c6bfab74408 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:21:55.200109 17754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1e4e725d37c4ee3bac78c6bfab74408 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:55.202775 17754 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1e4e725d37c4ee3bac78c6bfab74408 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e4e725d37c4ee3bac78c6bfab74408" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 35353 } }
I20260929 20:21:55.203285 17754 leader_election.cc:302] T 00000000000000000000000000000000 P e1e4e725d37c4ee3bac78c6bfab74408 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1e4e725d37c4ee3bac78c6bfab74408; no voters:
I20260929 20:21:55.204102 17754 leader_election.cc:288] T 00000000000000000000000000000000 P e1e4e725d37c4ee3bac78c6bfab74408 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:21:55.204159 17764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1e4e725d37c4ee3bac78c6bfab74408 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:21:55.206355 17754 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1e4e725d37c4ee3bac78c6bfab74408 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:21:55.207710 17764 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1e4e725d37c4ee3bac78c6bfab74408 [term 1 LEADER]: Becoming Leader. State: Replica: e1e4e725d37c4ee3bac78c6bfab74408, State: Running, Role: LEADER
I20260929 20:21:55.208477 17764 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1e4e725d37c4ee3bac78c6bfab74408 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e4e725d37c4ee3bac78c6bfab74408" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 35353 } }
I20260929 20:21:55.211891 17766 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e4e725d37c4ee3bac78c6bfab74408 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1e4e725d37c4ee3bac78c6bfab74408. Latest consensus state: current_term: 1 leader_uuid: "e1e4e725d37c4ee3bac78c6bfab74408" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e4e725d37c4ee3bac78c6bfab74408" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 35353 } } }
I20260929 20:21:55.212709 17766 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e4e725d37c4ee3bac78c6bfab74408 [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:55.212674 17765 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e4e725d37c4ee3bac78c6bfab74408 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1e4e725d37c4ee3bac78c6bfab74408" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e4e725d37c4ee3bac78c6bfab74408" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 35353 } } }
I20260929 20:21:55.213786 17765 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e4e725d37c4ee3bac78c6bfab74408 [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:55.214511 17774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:21:55.222558 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:55.201011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44259 (local address 127.27.185.126:35353)
0929 20:21:55.201208 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:55.201219 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:55.201688 (+ 469us) server_negotiation.cc:407] Connection header received
0929 20:21:55.201810 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:55.201819 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:55.201895 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:55.202005 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:55.203481 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:55.204814 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:55.206631 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:55.207313 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:55.207553 (+ 240us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:55.211188 (+ 3635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:55.211200 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:55.211206 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:21:55.211263 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:55.217278 (+ 6015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:55.218640 (+ 1362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:55.218649 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:55.218655 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:55.218864 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:21:55.219645 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:21:55.219652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:21:55.219658 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:21:55.220206 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:55.221846 (+ 1640us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:55.222033 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 20:21:55.222203 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":59}
I20260929 20:21:55.225276 17774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:21:55.235301 17774 catalog_manager.cc:1347] Generated new cluster ID: 749969b9fb814aaca4f8b3c4355e6099
I20260929 20:21:55.235457 17774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:21:55.247956 17774 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:21:55.249423 17774 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:21:55.259196 17774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1e4e725d37c4ee3bac78c6bfab74408: Generated new TSK 0
I20260929 20:21:55.260150 17774 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:21:55.269896 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 17687
I20260929 20:21:55.271363 17749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:21:55.362704 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 17687
W20260929 20:21:55.379976 17672 connection.cc:535] client connection to 127.27.185.126:35353 recv error: Network error: failed to read from TLS socket (remote: 127.27.185.126:35353): Connection reset by peer (error 104)
2026-09-29T20:21:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2048 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:21:55 dist-test-slave-4cx3 krb5kdc[17786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:21:55 dist-test-slave-4cx3 krb5kdc[17786](info): set up 2 sockets
Sep 29 20:21:55 dist-test-slave-4cx3 krb5kdc[17786](info): commencing operation
krb5kdc: starting...
W20260929 20:21:56.478812 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:21:56 dist-test-slave-4cx3 krb5kdc[17786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:21:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:21:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
I20260929 20:21:56.746757 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:45701
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:39553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:45701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:21:57.087543 17799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:57.087837 17799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:57.088020 17799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:57.095309 17799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:21:57.095475 17799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:57.095564 17799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:57.095672 17799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:21:57.095736 17799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:21:57.105600 17799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:39553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:45701
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:45701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:57.106911 17799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:57.108263 17799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:57.120267 17807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:57.120334 17804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:57.120303 17805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:57.121136 17799 server_base.cc:1034] running on GCE node
I20260929 20:21:57.121960 17799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:57.123524 17799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:57.124729 17799 hybrid_clock.cc:648] HybridClock initialized: now 1790713317124674 us; error 49 us; skew 500 ppm
I20260929 20:21:57.125090 17799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:21:57 dist-test-slave-4cx3 krb5kdc[17786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713317, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:21:57.128484 17799 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:21:57.130057 17799 webserver.cc:460] Webserver started at http://127.27.185.126:45573/ using document root <none> and password file <none>
I20260929 20:21:57.130792 17799 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:57.130934 17799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:57.131271 17799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:57.134524 17799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "0261a85ccda246bcb5001e9e1f8006fa"
format_stamp: "Formatted at 2026-09-29 20:21:57 on dist-test-slave-4cx3"
server_key: "ffb7f2d8f953d06cd64015085e3f73de"
server_key_iv: "76b56c49b57d72971ef00f2e4fd185ec"
server_key_version: "encryptionkey@0"
I20260929 20:21:57.135505 17799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "0261a85ccda246bcb5001e9e1f8006fa"
format_stamp: "Formatted at 2026-09-29 20:21:57 on dist-test-slave-4cx3"
server_key: "ffb7f2d8f953d06cd64015085e3f73de"
server_key_iv: "76b56c49b57d72971ef00f2e4fd185ec"
server_key_version: "encryptionkey@0"
I20260929 20:21:57.140856 17799 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:21:57.144488 17814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:57.145859 17799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:21:57.146055 17799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "0261a85ccda246bcb5001e9e1f8006fa"
format_stamp: "Formatted at 2026-09-29 20:21:57 on dist-test-slave-4cx3"
server_key: "ffb7f2d8f953d06cd64015085e3f73de"
server_key_iv: "76b56c49b57d72971ef00f2e4fd185ec"
server_key_version: "encryptionkey@0"
I20260929 20:21:57.146267 17799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:57.160470 17799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:57.165395 17799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:57.165714 17799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:57.185086 17799 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:45701
I20260929 20:21:57.185105 17865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:45701 every 8 connection(s)
I20260929 20:21:57.186580 17799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:21:57.190361 17866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:57.193470 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 17799
I20260929 20:21:57.193666 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:21:57.194118 28389 external_mini_cluster.cc:1428] Setting key d59dd8f2d379fa46fc6a3f22741559f4
I20260929 20:21:57.203799 17866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0261a85ccda246bcb5001e9e1f8006fa: Bootstrap starting.
I20260929 20:21:57.207662 17866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0261a85ccda246bcb5001e9e1f8006fa: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:57.209086 17866 log.cc:824] T 00000000000000000000000000000000 P 0261a85ccda246bcb5001e9e1f8006fa: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:57.209892 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:57.196067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42555 (local address 127.27.185.126:45701)
0929 20:21:57.196843 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:57.196859 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:57.196907 (+ 48us) server_negotiation.cc:407] Connection header received
0929 20:21:57.199194 (+ 2287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:57.199224 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:57.199804 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:57.200237 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:57.200770 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:57.202772 (+ 2002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:57.204359 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:57.205380 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:57.205624 (+ 244us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:57.206913 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:57.206924 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:57.206943 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:57.207681 (+ 738us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:57.207781 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:57.207819 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:57.207991 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:57.208306 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 20:21:57.208477 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":330,"threads_started":1}
I20260929 20:21:57.212170 17866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0261a85ccda246bcb5001e9e1f8006fa: No bootstrap required, opened a new log
I20260929 20:21:57.213840 28389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:21:57.222512 17866 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0261a85ccda246bcb5001e9e1f8006fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0261a85ccda246bcb5001e9e1f8006fa" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 45701 } }
I20260929 20:21:57.222982 17866 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0261a85ccda246bcb5001e9e1f8006fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:57.223095 17866 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0261a85ccda246bcb5001e9e1f8006fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0261a85ccda246bcb5001e9e1f8006fa, State: Initialized, Role: FOLLOWER
I20260929 20:21:57.223757 17866 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0261a85ccda246bcb5001e9e1f8006fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0261a85ccda246bcb5001e9e1f8006fa" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 45701 } }
I20260929 20:21:57.224014 17866 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0261a85ccda246bcb5001e9e1f8006fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:21:57.224193 17866 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0261a85ccda246bcb5001e9e1f8006fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:21:57.224385 17866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0261a85ccda246bcb5001e9e1f8006fa [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:57.226262 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:57.219025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49973 (local address 127.27.185.126:45701)
0929 20:21:57.219236 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:57.219244 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:57.219790 (+ 546us) server_negotiation.cc:407] Connection header received
0929 20:21:57.220016 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:57.220024 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:57.220097 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:57.220190 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:57.220812 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:57.221602 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:57.223430 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:21:57.224007 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:21:57.224150 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:21:57.225470 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:57.225476 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:57.225480 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:57.225498 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:57.225558 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:57.225576 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:57.225710 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:57.225928 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 20:21:57.225984 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:21:57.227294 17866 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0261a85ccda246bcb5001e9e1f8006fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0261a85ccda246bcb5001e9e1f8006fa" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 45701 } }
I20260929 20:21:57.227864 17866 leader_election.cc:302] T 00000000000000000000000000000000 P 0261a85ccda246bcb5001e9e1f8006fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0261a85ccda246bcb5001e9e1f8006fa; no voters:
I20260929 20:21:57.228477 17866 leader_election.cc:288] T 00000000000000000000000000000000 P 0261a85ccda246bcb5001e9e1f8006fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:21:57.228626 17876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0261a85ccda246bcb5001e9e1f8006fa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:21:57.229421 17876 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0261a85ccda246bcb5001e9e1f8006fa [term 1 LEADER]: Becoming Leader. State: Replica: 0261a85ccda246bcb5001e9e1f8006fa, State: Running, Role: LEADER
I20260929 20:21:57.230046 17876 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0261a85ccda246bcb5001e9e1f8006fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0261a85ccda246bcb5001e9e1f8006fa" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 45701 } }
I20260929 20:21:57.230801 17866 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0261a85ccda246bcb5001e9e1f8006fa [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:21:57.233314 17878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0261a85ccda246bcb5001e9e1f8006fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0261a85ccda246bcb5001e9e1f8006fa. Latest consensus state: current_term: 1 leader_uuid: "0261a85ccda246bcb5001e9e1f8006fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0261a85ccda246bcb5001e9e1f8006fa" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 45701 } } }
I20260929 20:21:57.233738 17878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0261a85ccda246bcb5001e9e1f8006fa [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:57.233810 17877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0261a85ccda246bcb5001e9e1f8006fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0261a85ccda246bcb5001e9e1f8006fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0261a85ccda246bcb5001e9e1f8006fa" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 45701 } } }
I20260929 20:21:57.234098 17877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0261a85ccda246bcb5001e9e1f8006fa [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:57.237342 17885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:21:57.242031 17885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:21:57.251209 17885 catalog_manager.cc:1347] Generated new cluster ID: a1a312f797b54d2a99fda971fd7b3146
I20260929 20:21:57.251336 17885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:21:57.269235 17885 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:21:57.271154 17885 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:21:57.284191 17885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0261a85ccda246bcb5001e9e1f8006fa: Generated new TSK 0
I20260929 20:21:57.285317 17885 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:21:57.291435 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 17799
W20260929 20:21:57.292975 17861 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42555
W20260929 20:21:57.293733 28389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 17799: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:21:57.293883 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 17799
W20260929 20:21:57.307672 17784 connection.cc:535] client connection to 127.27.185.126:45701 recv error: Network error: failed to read from TLS socket (remote: 127.27.185.126:45701): Connection reset by peer (error 104)
2026-09-29T20:21:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1927 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:21:57 dist-test-slave-4cx3 krb5kdc[17897](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:21:57 dist-test-slave-4cx3 krb5kdc[17897](info): set up 2 sockets
Sep 29 20:21:57 dist-test-slave-4cx3 krb5kdc[17897](info): commencing operation
krb5kdc: starting...
W20260929 20:21:58.402889 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:21:58 dist-test-slave-4cx3 krb5kdc[17897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:21:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:21:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
I20260929 20:21:58.670843 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:41099
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:40491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:41099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:21:59.008661 17910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:21:59.009049 17910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:21:59.009217 17910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:21:59.016855 17910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:21:59.017030 17910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:21:59.017145 17910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:21:59.017256 17910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:21:59.017335 17910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:21:59.027611 17910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:40491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:41099
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:41099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:21:59.029006 17910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:21:59.030381 17910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:21:59.042627 17918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:59.042645 17915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:21:59.042676 17916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:21:59.043565 17910 server_base.cc:1034] running on GCE node
I20260929 20:21:59.044389 17910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:21:59.046061 17910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:21:59.047317 17910 hybrid_clock.cc:648] HybridClock initialized: now 1790713319047252 us; error 50 us; skew 500 ppm
I20260929 20:21:59.047730 17910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:21:59 dist-test-slave-4cx3 krb5kdc[17897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713319, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:21:59.055863 17910 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:21:59.057575 17910 webserver.cc:460] Webserver started at http://127.27.185.126:46547/ using document root <none> and password file <none>
I20260929 20:21:59.058347 17910 fs_manager.cc:359] Metadata directory not provided
I20260929 20:21:59.058501 17910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:21:59.058820 17910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:21:59.062033 17910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "de4a43adf9054ee1b374083a710b890e"
format_stamp: "Formatted at 2026-09-29 20:21:59 on dist-test-slave-4cx3"
server_key: "9fd84cbca7479312af4f8645430849a3"
server_key_iv: "54dcd553e71110338f51084d24fb2acc"
server_key_version: "encryptionkey@0"
I20260929 20:21:59.062852 17910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "de4a43adf9054ee1b374083a710b890e"
format_stamp: "Formatted at 2026-09-29 20:21:59 on dist-test-slave-4cx3"
server_key: "9fd84cbca7479312af4f8645430849a3"
server_key_iv: "54dcd553e71110338f51084d24fb2acc"
server_key_version: "encryptionkey@0"
I20260929 20:21:59.068383 17910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 20:21:59.072285 17925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:21:59.073719 17910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:21:59.073925 17910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "de4a43adf9054ee1b374083a710b890e"
format_stamp: "Formatted at 2026-09-29 20:21:59 on dist-test-slave-4cx3"
server_key: "9fd84cbca7479312af4f8645430849a3"
server_key_iv: "54dcd553e71110338f51084d24fb2acc"
server_key_version: "encryptionkey@0"
I20260929 20:21:59.074148 17910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:21:59.080399 17910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:21:59.084642 17910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:21:59.084961 17910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:21:59.104597 17910 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:41099
I20260929 20:21:59.104620 17976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:41099 every 8 connection(s)
I20260929 20:21:59.106112 17910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:21:59.106681 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 17910
I20260929 20:21:59.106899 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:21:59.107558 28389 external_mini_cluster.cc:1428] Setting key b5f266968d6db9388565ac6f69226389
I20260929 20:21:59.110831 17977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:21:59.115479 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:59.109428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43431 (local address 127.27.185.126:41099)
0929 20:21:59.110210 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:59.110223 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:59.110268 (+ 45us) server_negotiation.cc:407] Connection header received
0929 20:21:59.112037 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:59.112059 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:59.112390 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:59.112773 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:59.113824 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:59.113830 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:59.113842 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:59.113865 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:59.113939 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:59.113957 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:59.114191 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:59.114335 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 20:21:59.114513 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":334,"threads_started":1}
I20260929 20:21:59.119365 28389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:21:59.122709 17977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de4a43adf9054ee1b374083a710b890e: Bootstrap starting.
I20260929 20:21:59.126322 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:21:59.123759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35653 (local address 127.27.185.126:41099)
0929 20:21:59.123938 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:21:59.123944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:21:59.124442 (+ 498us) server_negotiation.cc:407] Connection header received
0929 20:21:59.124592 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:21:59.124597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:21:59.124664 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:21:59.124777 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:21:59.125696 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:21:59.125702 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:21:59.125706 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:21:59.125726 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:21:59.125773 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:21:59.125794 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:21:59.125906 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 20:21:59.125987 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 20:21:59.126042 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:21:59.126497 17977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de4a43adf9054ee1b374083a710b890e: Neither blocks nor log segments found. Creating new log.
I20260929 20:21:59.127709 17977 log.cc:824] T 00000000000000000000000000000000 P de4a43adf9054ee1b374083a710b890e: Log is configured to *not* fsync() on all Append() calls
I20260929 20:21:59.130599 17977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de4a43adf9054ee1b374083a710b890e: No bootstrap required, opened a new log
I20260929 20:21:59.138322 17977 raft_consensus.cc:348] T 00000000000000000000000000000000 P de4a43adf9054ee1b374083a710b890e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4a43adf9054ee1b374083a710b890e" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41099 } }
I20260929 20:21:59.138767 17977 raft_consensus.cc:374] T 00000000000000000000000000000000 P de4a43adf9054ee1b374083a710b890e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:21:59.138880 17977 raft_consensus.cc:729] T 00000000000000000000000000000000 P de4a43adf9054ee1b374083a710b890e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de4a43adf9054ee1b374083a710b890e, State: Initialized, Role: FOLLOWER
I20260929 20:21:59.139554 17977 consensus_queue.cc:260] T 00000000000000000000000000000000 P de4a43adf9054ee1b374083a710b890e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4a43adf9054ee1b374083a710b890e" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41099 } }
I20260929 20:21:59.139881 17977 raft_consensus.cc:388] T 00000000000000000000000000000000 P de4a43adf9054ee1b374083a710b890e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:21:59.140039 17977 raft_consensus.cc:482] T 00000000000000000000000000000000 P de4a43adf9054ee1b374083a710b890e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:21:59.140221 17977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de4a43adf9054ee1b374083a710b890e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:21:59.142782 17977 raft_consensus.cc:504] T 00000000000000000000000000000000 P de4a43adf9054ee1b374083a710b890e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4a43adf9054ee1b374083a710b890e" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41099 } }
I20260929 20:21:59.143309 17977 leader_election.cc:302] T 00000000000000000000000000000000 P de4a43adf9054ee1b374083a710b890e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de4a43adf9054ee1b374083a710b890e; no voters:
I20260929 20:21:59.143931 17977 leader_election.cc:288] T 00000000000000000000000000000000 P de4a43adf9054ee1b374083a710b890e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:21:59.144088 17987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de4a43adf9054ee1b374083a710b890e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:21:59.144796 17987 raft_consensus.cc:686] T 00000000000000000000000000000000 P de4a43adf9054ee1b374083a710b890e [term 1 LEADER]: Becoming Leader. State: Replica: de4a43adf9054ee1b374083a710b890e, State: Running, Role: LEADER
I20260929 20:21:59.145315 17987 consensus_queue.cc:237] T 00000000000000000000000000000000 P de4a43adf9054ee1b374083a710b890e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4a43adf9054ee1b374083a710b890e" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41099 } }
I20260929 20:21:59.146119 17977 sys_catalog.cc:564] T 00000000000000000000000000000000 P de4a43adf9054ee1b374083a710b890e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:21:59.148608 17988 sys_catalog.cc:455] T 00000000000000000000000000000000 P de4a43adf9054ee1b374083a710b890e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de4a43adf9054ee1b374083a710b890e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4a43adf9054ee1b374083a710b890e" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41099 } } }
I20260929 20:21:59.148703 17989 sys_catalog.cc:455] T 00000000000000000000000000000000 P de4a43adf9054ee1b374083a710b890e [sys.catalog]: SysCatalogTable state changed. Reason: New leader de4a43adf9054ee1b374083a710b890e. Latest consensus state: current_term: 1 leader_uuid: "de4a43adf9054ee1b374083a710b890e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4a43adf9054ee1b374083a710b890e" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 41099 } } }
I20260929 20:21:59.149021 17988 sys_catalog.cc:458] T 00000000000000000000000000000000 P de4a43adf9054ee1b374083a710b890e [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:59.149091 17989 sys_catalog.cc:458] T 00000000000000000000000000000000 P de4a43adf9054ee1b374083a710b890e [sys.catalog]: This master's current role is: LEADER
I20260929 20:21:59.151556 17993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:21:59.157164 17993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:21:59.166121 17993 catalog_manager.cc:1347] Generated new cluster ID: e5d91adecf6a4b3491269d3a3934a91d
I20260929 20:21:59.166265 17993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:21:59.186743 17993 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:21:59.187228 17993 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:21:59.204609 17993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de4a43adf9054ee1b374083a710b890e: Generated new TSK 0
I20260929 20:21:59.205487 17993 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:21:59.214370 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 17910
W20260929 20:21:59.216001 17972 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43431
W20260929 20:21:59.216768 28389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 17910: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:21:59.216917 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 17910
2026-09-29T20:21:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1923 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:21:59 dist-test-slave-4cx3 krb5kdc[18009](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:21:59 dist-test-slave-4cx3 krb5kdc[18009](info): set up 2 sockets
Sep 29 20:21:59 dist-test-slave-4cx3 krb5kdc[18009](info): commencing operation
krb5kdc: starting...
W20260929 20:22:00.314831 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:22:00 dist-test-slave-4cx3 krb5kdc[18009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:22:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:22:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
I20260929 20:22:00.584312 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:34433
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:44565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:34433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:00.923122 18022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:00.923487 18022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:00.923673 18022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:00.931058 18022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:22:00.931198 18022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:00.931311 18022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:00.931439 18022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:22:00.931504 18022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:22:00.941680 18022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:44565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:34433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:34433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:00.943027 18022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:00.944500 18022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:00.956702 18027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:00.956743 18030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:00.956777 18028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:00.957360 18022 server_base.cc:1034] running on GCE node
I20260929 20:22:00.958154 18022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:00.959661 18022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:00.960897 18022 hybrid_clock.cc:648] HybridClock initialized: now 1790713320960860 us; error 45 us; skew 500 ppm
I20260929 20:22:00.961261 18022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:00 dist-test-slave-4cx3 krb5kdc[18009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713320, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:00.964372 18022 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:22:00.965827 18022 webserver.cc:460] Webserver started at http://127.27.185.126:35807/ using document root <none> and password file <none>
I20260929 20:22:00.966485 18022 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:00.966590 18022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:00.966840 18022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:00.969918 18022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "67b7a61552c14c439ff192ad93f72d97"
format_stamp: "Formatted at 2026-09-29 20:22:00 on dist-test-slave-4cx3"
server_key: "257e3bd8a6b2228347ef529f1a2b6da4"
server_key_iv: "921f79a23a6fd58aa8c3a74468d26b80"
server_key_version: "encryptionkey@0"
I20260929 20:22:00.970649 18022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "67b7a61552c14c439ff192ad93f72d97"
format_stamp: "Formatted at 2026-09-29 20:22:00 on dist-test-slave-4cx3"
server_key: "257e3bd8a6b2228347ef529f1a2b6da4"
server_key_iv: "921f79a23a6fd58aa8c3a74468d26b80"
server_key_version: "encryptionkey@0"
I20260929 20:22:00.975914 18022 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:22:00.979496 18037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:00.980862 18022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:22:00.981063 18022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "67b7a61552c14c439ff192ad93f72d97"
format_stamp: "Formatted at 2026-09-29 20:22:00 on dist-test-slave-4cx3"
server_key: "257e3bd8a6b2228347ef529f1a2b6da4"
server_key_iv: "921f79a23a6fd58aa8c3a74468d26b80"
server_key_version: "encryptionkey@0"
I20260929 20:22:00.981268 18022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:01.008174 18022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:01.012313 18022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:01.012619 18022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:01.031445 18022 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:34433
I20260929 20:22:01.031504 18089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:34433 every 8 connection(s)
I20260929 20:22:01.032902 18022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:22:01.036936 18090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:01.040726 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 18022
I20260929 20:22:01.040913 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:22:01.041479 28389 external_mini_cluster.cc:1428] Setting key 0f5411f28c9808a96dc578b53001478e
I20260929 20:22:01.050709 18090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67b7a61552c14c439ff192ad93f72d97: Bootstrap starting.
I20260929 20:22:01.054639 18090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67b7a61552c14c439ff192ad93f72d97: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:01.056079 18090 log.cc:824] T 00000000000000000000000000000000 P 67b7a61552c14c439ff192ad93f72d97: Log is configured to *not* fsync() on all Append() calls
Sep 29 20:22:01 dist-test-slave-4cx3 krb5kdc[18009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:22:01.059584 18090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67b7a61552c14c439ff192ad93f72d97: No bootstrap required, opened a new log
I20260929 20:22:01.066205 18093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:01.043648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59505 (local address 127.27.185.126:34433)
0929 20:22:01.044628 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:01.044640 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:01.044685 (+ 45us) server_negotiation.cc:407] Connection header received
0929 20:22:01.046468 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:01.046491 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:01.046956 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:01.047328 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:01.049152 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:01.050569 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:01.052401 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:01.053084 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:01.053211 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:01.058785 (+ 5574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:01.058793 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:01.058808 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:01.058842 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:01.062055 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:01.062912 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:01.062918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:01.062925 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:01.063091 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:01.063515 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:01.063518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:01.063520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:01.064023 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:01.064254 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:01.064728 (+ 474us) server_negotiation.cc:299] Negotiation successful
0929 20:22:01.065145 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":753,"thread_start_us":474,"threads_started":1}
I20260929 20:22:01.068902 18090 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67b7a61552c14c439ff192ad93f72d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b7a61552c14c439ff192ad93f72d97" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34433 } }
I20260929 20:22:01.069337 18090 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67b7a61552c14c439ff192ad93f72d97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:01.069501 18090 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67b7a61552c14c439ff192ad93f72d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b7a61552c14c439ff192ad93f72d97, State: Initialized, Role: FOLLOWER
I20260929 20:22:01.070178 28389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:22:01.070169 18090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67b7a61552c14c439ff192ad93f72d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b7a61552c14c439ff192ad93f72d97" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34433 } }
I20260929 20:22:01.070474 18090 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67b7a61552c14c439ff192ad93f72d97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:22:01.070616 18090 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67b7a61552c14c439ff192ad93f72d97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:22:01.070787 18090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67b7a61552c14c439ff192ad93f72d97 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:01.073133 18090 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67b7a61552c14c439ff192ad93f72d97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b7a61552c14c439ff192ad93f72d97" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34433 } }
I20260929 20:22:01.073712 18090 leader_election.cc:302] T 00000000000000000000000000000000 P 67b7a61552c14c439ff192ad93f72d97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67b7a61552c14c439ff192ad93f72d97; no voters:
I20260929 20:22:01.074375 18090 leader_election.cc:288] T 00000000000000000000000000000000 P 67b7a61552c14c439ff192ad93f72d97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:22:01.074496 18099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67b7a61552c14c439ff192ad93f72d97 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:22:01.078761 18099 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67b7a61552c14c439ff192ad93f72d97 [term 1 LEADER]: Becoming Leader. State: Replica: 67b7a61552c14c439ff192ad93f72d97, State: Running, Role: LEADER
I20260929 20:22:01.079838 18099 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67b7a61552c14c439ff192ad93f72d97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b7a61552c14c439ff192ad93f72d97" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34433 } }
I20260929 20:22:01.081501 18090 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67b7a61552c14c439ff192ad93f72d97 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:22:01.082913 18101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67b7a61552c14c439ff192ad93f72d97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67b7a61552c14c439ff192ad93f72d97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b7a61552c14c439ff192ad93f72d97" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34433 } } }
I20260929 20:22:01.083376 18101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67b7a61552c14c439ff192ad93f72d97 [sys.catalog]: This master's current role is: LEADER
I20260929 20:22:01.083658 18100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67b7a61552c14c439ff192ad93f72d97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67b7a61552c14c439ff192ad93f72d97. Latest consensus state: current_term: 1 leader_uuid: "67b7a61552c14c439ff192ad93f72d97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b7a61552c14c439ff192ad93f72d97" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34433 } } }
I20260929 20:22:01.084621 18100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67b7a61552c14c439ff192ad93f72d97 [sys.catalog]: This master's current role is: LEADER
I20260929 20:22:01.091219 18093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:01.074807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55357 (local address 127.27.185.126:34433)
0929 20:22:01.074996 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:01.075005 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:01.075631 (+ 626us) server_negotiation.cc:407] Connection header received
0929 20:22:01.075844 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:01.075849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:01.075925 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:01.076039 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:01.077495 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:01.078363 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:01.080021 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:01.080538 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:01.080623 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:01.083888 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:01.083896 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:01.083901 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:01.083944 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:01.087896 (+ 3952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:01.088612 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:01.088617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:01.088620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:01.088768 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:01.089184 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:01.089188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:01.089190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:01.089676 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:01.090703 (+ 1027us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:01.090819 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 20:22:01.090984 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:22:01.096391 18105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:22:01.101737 18105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:22:01.111253 18105 catalog_manager.cc:1347] Generated new cluster ID: 9b3931ec61e7418fb9b60bbaa036dbe3
I20260929 20:22:01.111403 18105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:22:01.141520 18105 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:22:01.142858 18105 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:22:01.150969 18105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67b7a61552c14c439ff192ad93f72d97: Generated new TSK 0
I20260929 20:22:01.151852 18105 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:22:01.162760 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 18022
I20260929 20:22:01.163900 18085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:22:01.259109 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 18022
2026-09-29T20:22:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2045 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:22:01 dist-test-slave-4cx3 krb5kdc[18123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:22:01 dist-test-slave-4cx3 krb5kdc[18123](info): set up 2 sockets
Sep 29 20:22:01 dist-test-slave-4cx3 krb5kdc[18123](info): commencing operation
krb5kdc: starting...
W20260929 20:22:02.370833 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:22:02 dist-test-slave-4cx3 krb5kdc[18123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:22:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:22:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
I20260929 20:22:02.642416 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:44227
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:36551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:44227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:02.983979 18136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:02.984282 18136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:02.984490 18136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:02.991778 18136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:22:02.991928 18136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:02.992020 18136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:02.992131 18136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:22:02.992197 18136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:22:03.002146 18136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:36551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:44227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:44227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:03.003603 18136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:03.004953 18136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:03.016741 18144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:03.016784 18141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:03.016810 18142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:03.017642 18136 server_base.cc:1034] running on GCE node
I20260929 20:22:03.018478 18136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:03.019979 18136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:03.021196 18136 hybrid_clock.cc:648] HybridClock initialized: now 1790713323021133 us; error 53 us; skew 500 ppm
I20260929 20:22:03.021569 18136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:03 dist-test-slave-4cx3 krb5kdc[18123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713323, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:03.024910 18136 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:22:03.026412 18136 webserver.cc:460] Webserver started at http://127.27.185.126:34201/ using document root <none> and password file <none>
I20260929 20:22:03.027135 18136 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:03.027287 18136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:03.027650 18136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:03.030716 18136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "117287bddb7744b99dad6e22ed039f5b"
format_stamp: "Formatted at 2026-09-29 20:22:03 on dist-test-slave-4cx3"
server_key: "df433c1761c05f01e53a065a8c2029dc"
server_key_iv: "2caab5537515ceda763cccf12ce296cd"
server_key_version: "encryptionkey@0"
I20260929 20:22:03.031577 18136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "117287bddb7744b99dad6e22ed039f5b"
format_stamp: "Formatted at 2026-09-29 20:22:03 on dist-test-slave-4cx3"
server_key: "df433c1761c05f01e53a065a8c2029dc"
server_key_iv: "2caab5537515ceda763cccf12ce296cd"
server_key_version: "encryptionkey@0"
I20260929 20:22:03.036809 18136 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 20:22:03.040627 18152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:03.042057 18136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:22:03.042191 18136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "117287bddb7744b99dad6e22ed039f5b"
format_stamp: "Formatted at 2026-09-29 20:22:03 on dist-test-slave-4cx3"
server_key: "df433c1761c05f01e53a065a8c2029dc"
server_key_iv: "2caab5537515ceda763cccf12ce296cd"
server_key_version: "encryptionkey@0"
I20260929 20:22:03.042357 18136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:03.067549 18136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:03.072873 18136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:03.073124 18136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:03.092515 18136 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:44227
I20260929 20:22:03.092540 18203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:44227 every 8 connection(s)
I20260929 20:22:03.093928 18136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:22:03.097685 18204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:03.099819 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 18136
I20260929 20:22:03.100003 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:22:03.100550 28389 external_mini_cluster.cc:1428] Setting key f569163d4bea752bcf102c70a60a03f6
I20260929 20:22:03.110663 18204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 117287bddb7744b99dad6e22ed039f5b: Bootstrap starting.
I20260929 20:22:03.112936 18207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:03.102357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57931 (local address 127.27.185.126:44227)
0929 20:22:03.103114 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:03.103126 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:03.103169 (+ 43us) server_negotiation.cc:407] Connection header received
0929 20:22:03.104749 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:03.104772 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:03.105158 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:03.105527 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:03.106049 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:03.107279 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:03.108589 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:03.109265 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:03.109469 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:03.110661 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:03.110669 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:03.110683 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:22:03.111261 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:03.111336 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:22:03.111361 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:03.111540 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:03.111763 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 20:22:03.112014 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":323,"threads_started":1}
I20260929 20:22:03.113925 18204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 117287bddb7744b99dad6e22ed039f5b: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:03.114982 18204 log.cc:824] T 00000000000000000000000000000000 P 117287bddb7744b99dad6e22ed039f5b: Log is configured to *not* fsync() on all Append() calls
I20260929 20:22:03.116499 28389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:22:03.117841 18204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 117287bddb7744b99dad6e22ed039f5b: No bootstrap required, opened a new log
I20260929 20:22:03.127493 18207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:03.120932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39077 (local address 127.27.185.126:44227)
0929 20:22:03.121149 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:03.121155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:03.121452 (+ 297us) server_negotiation.cc:407] Connection header received
0929 20:22:03.121671 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:03.121674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:03.121736 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:03.121812 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:03.122410 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:03.123134 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:03.125035 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:03.125475 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:03.125567 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:03.126731 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:03.126735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:03.126738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:22:03.126755 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:03.126789 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:22:03.126803 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:03.126868 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:03.127029 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 20:22:03.127098 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 20:22:03.127832 18204 raft_consensus.cc:348] T 00000000000000000000000000000000 P 117287bddb7744b99dad6e22ed039f5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "117287bddb7744b99dad6e22ed039f5b" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 44227 } }
I20260929 20:22:03.128297 18204 raft_consensus.cc:374] T 00000000000000000000000000000000 P 117287bddb7744b99dad6e22ed039f5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:03.128430 18204 raft_consensus.cc:729] T 00000000000000000000000000000000 P 117287bddb7744b99dad6e22ed039f5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 117287bddb7744b99dad6e22ed039f5b, State: Initialized, Role: FOLLOWER
I20260929 20:22:03.129027 18204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 117287bddb7744b99dad6e22ed039f5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "117287bddb7744b99dad6e22ed039f5b" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 44227 } }
I20260929 20:22:03.129271 18204 raft_consensus.cc:388] T 00000000000000000000000000000000 P 117287bddb7744b99dad6e22ed039f5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:22:03.129441 18204 raft_consensus.cc:482] T 00000000000000000000000000000000 P 117287bddb7744b99dad6e22ed039f5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:22:03.129637 18204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 117287bddb7744b99dad6e22ed039f5b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:03.132032 18204 raft_consensus.cc:504] T 00000000000000000000000000000000 P 117287bddb7744b99dad6e22ed039f5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "117287bddb7744b99dad6e22ed039f5b" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 44227 } }
I20260929 20:22:03.132551 18204 leader_election.cc:302] T 00000000000000000000000000000000 P 117287bddb7744b99dad6e22ed039f5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 117287bddb7744b99dad6e22ed039f5b; no voters:
I20260929 20:22:03.133136 18204 leader_election.cc:288] T 00000000000000000000000000000000 P 117287bddb7744b99dad6e22ed039f5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:22:03.133379 18214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 117287bddb7744b99dad6e22ed039f5b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:22:03.134089 18214 raft_consensus.cc:686] T 00000000000000000000000000000000 P 117287bddb7744b99dad6e22ed039f5b [term 1 LEADER]: Becoming Leader. State: Replica: 117287bddb7744b99dad6e22ed039f5b, State: Running, Role: LEADER
I20260929 20:22:03.134819 18214 consensus_queue.cc:237] T 00000000000000000000000000000000 P 117287bddb7744b99dad6e22ed039f5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "117287bddb7744b99dad6e22ed039f5b" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 44227 } }
I20260929 20:22:03.135497 18204 sys_catalog.cc:564] T 00000000000000000000000000000000 P 117287bddb7744b99dad6e22ed039f5b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:22:03.137844 18216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 117287bddb7744b99dad6e22ed039f5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 117287bddb7744b99dad6e22ed039f5b. Latest consensus state: current_term: 1 leader_uuid: "117287bddb7744b99dad6e22ed039f5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "117287bddb7744b99dad6e22ed039f5b" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 44227 } } }
I20260929 20:22:03.138190 18216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 117287bddb7744b99dad6e22ed039f5b [sys.catalog]: This master's current role is: LEADER
I20260929 20:22:03.138785 18215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 117287bddb7744b99dad6e22ed039f5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "117287bddb7744b99dad6e22ed039f5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "117287bddb7744b99dad6e22ed039f5b" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 44227 } } }
I20260929 20:22:03.139081 18215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 117287bddb7744b99dad6e22ed039f5b [sys.catalog]: This master's current role is: LEADER
I20260929 20:22:03.139231 18222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:22:03.146106 18222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:22:03.156409 18222 catalog_manager.cc:1347] Generated new cluster ID: 8f1fb18c84534a11ab14fc23cf10392d
I20260929 20:22:03.156569 18222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:22:03.179416 18222 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:22:03.180862 18222 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:22:03.188197 18222 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 117287bddb7744b99dad6e22ed039f5b: Generated new TSK 0
I20260929 20:22:03.189334 18222 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:22:03.196615 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 18136
W20260929 20:22:03.197794 18199 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57931
W20260929 20:22:03.198571 28389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 18136: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:22:03.198727 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 18136
2026-09-29T20:22:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1939 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:22:03 dist-test-slave-4cx3 krb5kdc[18236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:22:03 dist-test-slave-4cx3 krb5kdc[18236](info): set up 2 sockets
Sep 29 20:22:03 dist-test-slave-4cx3 krb5kdc[18236](info): commencing operation
krb5kdc: starting...
W20260929 20:22:04.319164 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.008s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:22:04 dist-test-slave-4cx3 krb5kdc[18236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:22:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:22:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
I20260929 20:22:04.587956 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:42999
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:34747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:42999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:04.925206 18249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:04.925588 18249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:04.925773 18249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:04.933374 18249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:22:04.933564 18249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:04.933668 18249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:04.933777 18249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:22:04.933852 18249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:22:04.944124 18249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:34747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:42999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:42999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:04.945559 18249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:04.947031 18249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:04.959671 18257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:04.959714 18255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:04.960019 18254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:04.961050 18249 server_base.cc:1034] running on GCE node
I20260929 20:22:04.961977 18249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:04.963591 18249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:04.964839 18249 hybrid_clock.cc:648] HybridClock initialized: now 1790713324964790 us; error 42 us; skew 500 ppm
I20260929 20:22:04.965219 18249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:04 dist-test-slave-4cx3 krb5kdc[18236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713324, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:04.974678 18249 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:22:04.976548 18249 webserver.cc:460] Webserver started at http://127.27.185.126:37735/ using document root <none> and password file <none>
I20260929 20:22:04.977317 18249 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:04.977478 18249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:04.977805 18249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:04.981123 18249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "19385282100d4cd58f8624dce7f2f3e7"
format_stamp: "Formatted at 2026-09-29 20:22:04 on dist-test-slave-4cx3"
server_key: "d49dcae9529df8a10b86e2bfbcb8dbd4"
server_key_iv: "5f6478667e799b0acaf6aa4d6597edda"
server_key_version: "encryptionkey@0"
I20260929 20:22:04.981976 18249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "19385282100d4cd58f8624dce7f2f3e7"
format_stamp: "Formatted at 2026-09-29 20:22:04 on dist-test-slave-4cx3"
server_key: "d49dcae9529df8a10b86e2bfbcb8dbd4"
server_key_iv: "5f6478667e799b0acaf6aa4d6597edda"
server_key_version: "encryptionkey@0"
I20260929 20:22:04.987852 18249 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260929 20:22:04.991663 18264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:04.993176 18249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 20:22:04.993399 18249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "19385282100d4cd58f8624dce7f2f3e7"
format_stamp: "Formatted at 2026-09-29 20:22:04 on dist-test-slave-4cx3"
server_key: "d49dcae9529df8a10b86e2bfbcb8dbd4"
server_key_iv: "5f6478667e799b0acaf6aa4d6597edda"
server_key_version: "encryptionkey@0"
I20260929 20:22:04.993630 18249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:04.999809 18249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:05.004518 18249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:05.004863 18249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:05.024257 18249 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:42999
I20260929 20:22:05.024287 18316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:42999 every 8 connection(s)
I20260929 20:22:05.025712 18249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:22:05.029490 18317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:05.033565 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 18249
I20260929 20:22:05.033758 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:22:05.034315 28389 external_mini_cluster.cc:1428] Setting key feb7e0c378b7d28b21acc8959692f1fe
I20260929 20:22:05.042347 18320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:05.036271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42059 (local address 127.27.185.126:42999)
0929 20:22:05.037136 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:05.037149 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:05.037197 (+ 48us) server_negotiation.cc:407] Connection header received
0929 20:22:05.038831 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:05.038856 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:05.039231 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:05.039644 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:05.040688 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:05.040695 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:05.040707 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:22:05.040721 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:05.040786 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:22:05.040807 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:05.040990 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:05.041157 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 20:22:05.041433 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":402,"threads_started":1}
I20260929 20:22:05.043082 18317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19385282100d4cd58f8624dce7f2f3e7: Bootstrap starting.
I20260929 20:22:05.046070 28389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:22:05.046558 18317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19385282100d4cd58f8624dce7f2f3e7: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:05.047704 18317 log.cc:824] T 00000000000000000000000000000000 P 19385282100d4cd58f8624dce7f2f3e7: Log is configured to *not* fsync() on all Append() calls
I20260929 20:22:05.051190 18317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19385282100d4cd58f8624dce7f2f3e7: No bootstrap required, opened a new log
I20260929 20:22:05.053267 18320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:05.050610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49913 (local address 127.27.185.126:42999)
0929 20:22:05.050790 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:05.050798 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:05.051159 (+ 361us) server_negotiation.cc:407] Connection header received
0929 20:22:05.051447 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:05.051456 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:05.051501 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:05.051607 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:05.052559 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:05.052570 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:05.052574 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:22:05.052612 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:05.052657 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:22:05.052680 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:05.052773 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:05.052928 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 20:22:05.052974 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 20:22:05.059481 18317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19385282100d4cd58f8624dce7f2f3e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19385282100d4cd58f8624dce7f2f3e7" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 42999 } }
I20260929 20:22:05.059861 18317 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19385282100d4cd58f8624dce7f2f3e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:05.059953 18317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19385282100d4cd58f8624dce7f2f3e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19385282100d4cd58f8624dce7f2f3e7, State: Initialized, Role: FOLLOWER
I20260929 20:22:05.060675 18317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19385282100d4cd58f8624dce7f2f3e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19385282100d4cd58f8624dce7f2f3e7" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 42999 } }
I20260929 20:22:05.060935 18317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19385282100d4cd58f8624dce7f2f3e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:22:05.061112 18317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19385282100d4cd58f8624dce7f2f3e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:22:05.061290 18317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19385282100d4cd58f8624dce7f2f3e7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:05.063673 18317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19385282100d4cd58f8624dce7f2f3e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19385282100d4cd58f8624dce7f2f3e7" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 42999 } }
I20260929 20:22:05.064155 18317 leader_election.cc:302] T 00000000000000000000000000000000 P 19385282100d4cd58f8624dce7f2f3e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19385282100d4cd58f8624dce7f2f3e7; no voters:
I20260929 20:22:05.064785 18317 leader_election.cc:288] T 00000000000000000000000000000000 P 19385282100d4cd58f8624dce7f2f3e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:22:05.064908 18327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19385282100d4cd58f8624dce7f2f3e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:22:05.065553 18327 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19385282100d4cd58f8624dce7f2f3e7 [term 1 LEADER]: Becoming Leader. State: Replica: 19385282100d4cd58f8624dce7f2f3e7, State: Running, Role: LEADER
I20260929 20:22:05.066139 18327 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19385282100d4cd58f8624dce7f2f3e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19385282100d4cd58f8624dce7f2f3e7" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 42999 } }
I20260929 20:22:05.067250 18317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19385282100d4cd58f8624dce7f2f3e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:22:05.069286 18329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19385282100d4cd58f8624dce7f2f3e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19385282100d4cd58f8624dce7f2f3e7. Latest consensus state: current_term: 1 leader_uuid: "19385282100d4cd58f8624dce7f2f3e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19385282100d4cd58f8624dce7f2f3e7" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 42999 } } }
I20260929 20:22:05.069720 18329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19385282100d4cd58f8624dce7f2f3e7 [sys.catalog]: This master's current role is: LEADER
I20260929 20:22:05.069614 18328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19385282100d4cd58f8624dce7f2f3e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19385282100d4cd58f8624dce7f2f3e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19385282100d4cd58f8624dce7f2f3e7" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 42999 } } }
I20260929 20:22:05.070151 18328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19385282100d4cd58f8624dce7f2f3e7 [sys.catalog]: This master's current role is: LEADER
I20260929 20:22:05.072157 18334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:22:05.079349 18334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:22:05.090054 18334 catalog_manager.cc:1347] Generated new cluster ID: 2b23c6cc56904f1ebfb67b742aeae153
I20260929 20:22:05.090183 18334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:22:05.101282 18334 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:22:05.101732 18334 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:22:05.109668 18334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19385282100d4cd58f8624dce7f2f3e7: Generated new TSK 0
I20260929 20:22:05.110534 18334 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:22:05.125890 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 18249
W20260929 20:22:05.127327 18312 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42059
W20260929 20:22:05.128105 28389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 18249: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:22:05.128248 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 18249
2026-09-29T20:22:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1929 ms)
[----------] 18 tests from AuthTokenIssuingTest (35575 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:22:05 dist-test-slave-4cx3 krb5kdc[18348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:22:05 dist-test-slave-4cx3 krb5kdc[18348](info): set up 2 sockets
Sep 29 20:22:05 dist-test-slave-4cx3 krb5kdc[18348](info): commencing operation
krb5kdc: starting...
W20260929 20:22:06.235114 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:22:06 dist-test-slave-4cx3 krb5kdc[18348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:22:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:22:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
I20260929 20:22:06.507706 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:34175
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:34175,127.27.185.125:45305,127.27.185.124:33429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:06.851673 18361 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 20:22:06.852034 18361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:06.852155 18361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:06.852296 18361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:06.859845 18361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:22:06.859936 18361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:06.859992 18361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:06.860036 18361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:22:06.860096 18361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:22:06.871297 18361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:34175,127.27.185.125:45305,127.27.185.124:33429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:34175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:06.872696 18361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:06.874017 18361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:06.886276 18366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:06.886322 18369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:06.886327 18367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:06.887457 18361 server_base.cc:1034] running on GCE node
I20260929 20:22:06.888285 18361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:06.889753 18361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:06.890980 18361 hybrid_clock.cc:648] HybridClock initialized: now 1790713326890912 us; error 34 us; skew 500 ppm
I20260929 20:22:06.891364 18361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:06 dist-test-slave-4cx3 krb5kdc[18348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713326, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:06.894754 18361 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:22:06.896363 18361 webserver.cc:460] Webserver started at http://127.27.185.126:34353/ using document root <none> and password file <none>
I20260929 20:22:06.897081 18361 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:06.897197 18361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:06.897521 18361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:06.900544 18361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "07eb795753c540b082eb53db45b9c522"
format_stamp: "Formatted at 2026-09-29 20:22:06 on dist-test-slave-4cx3"
server_key: "791e9fc8e654753e5796c2c916d3612a"
server_key_iv: "1c5c8ae966793c9436e1926899a55a5f"
server_key_version: "encryptionkey@0"
I20260929 20:22:06.901333 18361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "07eb795753c540b082eb53db45b9c522"
format_stamp: "Formatted at 2026-09-29 20:22:06 on dist-test-slave-4cx3"
server_key: "791e9fc8e654753e5796c2c916d3612a"
server_key_iv: "1c5c8ae966793c9436e1926899a55a5f"
server_key_version: "encryptionkey@0"
I20260929 20:22:06.906971 18361 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:22:06.910570 18376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:06.911978 18361 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:22:06.912161 18361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "07eb795753c540b082eb53db45b9c522"
format_stamp: "Formatted at 2026-09-29 20:22:06 on dist-test-slave-4cx3"
server_key: "791e9fc8e654753e5796c2c916d3612a"
server_key_iv: "1c5c8ae966793c9436e1926899a55a5f"
server_key_version: "encryptionkey@0"
I20260929 20:22:06.912379 18361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:06.929680 18361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:06.933765 18361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:06.934051 18361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:06.952322 18361 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:34175
I20260929 20:22:06.952340 18427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:34175 every 8 connection(s)
I20260929 20:22:06.953699 18361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:22:06.957479 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 18361
I20260929 20:22:06.957707 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:22:06.957801 18428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:06.958556 28389 external_mini_cluster.cc:1428] Setting key 5334b5e2cc7e5f147dbce8e33cf94b00
I20260929 20:22:06.968194 18428 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34175 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:22:06 dist-test-slave-4cx3 krb5kdc[18348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713326, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.185.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.125@KRBTEST.COM" created.
I20260929 20:22:06.994059 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:06.972073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39724 (local address 127.27.185.126:34175)
0929 20:22:06.973090 (+ 1017us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:06.973112 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:06.973880 (+ 768us) server_negotiation.cc:407] Connection header received
0929 20:22:06.976028 (+ 2148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:06.976039 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:06.976360 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:06.976594 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:06.977782 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:06.979813 (+ 2031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:06.981205 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:06.982222 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:06.982423 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:06.986092 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:06.986105 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:06.986124 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:06.986167 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:06.989360 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:06.990189 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:06.990202 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:06.990204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:06.990362 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:06.990751 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:06.990758 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:06.990762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:06.991281 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:06.991515 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:06.992111 (+ 596us) server_negotiation.cc:299] Negotiation successful
0929 20:22:06.992514 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":725,"thread_start_us":393,"threads_started":1}
I20260929 20:22:06.994078 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:06.972072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:34175 (local address 127.0.0.1:39724)
0929 20:22:06.973666 (+ 1594us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:06.973716 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:06.975621 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:06.976619 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:06.976625 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:06.977199 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:06.977558 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:06.977575 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:06.979964 (+ 2389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:06.979968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:06.981056 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:06.981068 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:06.982456 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:06.982462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:06.982587 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:06.983283 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:06.983307 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:06.985831 (+ 2524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:06.989499 (+ 3668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:06.989510 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:06.989520 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:06.990045 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:06.990475 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:06.990482 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:06.990485 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:06.990643 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:06.991462 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:06.991469 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:06.991705 (+ 236us) client_negotiation.cc:769] Sending connection context
0929 20:22:06.991977 (+ 272us) client_negotiation.cc:240] Negotiation successful
0929 20:22:06.992632 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1334,"mutex_wait_us":155,"thread_start_us":1137,"threads_started":1}
I20260929 20:22:06.997188 18428 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.125" port: 45305 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:22:06.998499 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:06.998090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:45305 (local address 127.0.0.1:37056)
0929 20:22:06.998257 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:06.998361 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.125:45305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 20:22:06.998782 18380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.185.125:45305: connect: Connection refused (error 111)
I20260929 20:22:07.000555 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.000062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:45305 (local address 127.0.0.1:37066)
0929 20:22:07.000275 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:07.000370 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.125:45305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 20:22:07.000828 18428 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.125:45305: Network error: Client connection negotiation failed: client connection to 127.27.185.125:45305: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.125.keytab.
Entry for principal kudu/127.27.185.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.125.keytab.
I20260929 20:22:07.022439 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.125:45305
--webserver_interface=127.27.185.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.125.keytab
--principal=kudu/127.27.185.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:34175,127.27.185.125:45305,127.27.185.124:33429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:22:07.050195 18428 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.125" port: 45305 } attempt: 1
I20260929 20:22:07.051571 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.050931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:45305 (local address 127.0.0.1:37076)
0929 20:22:07.051166 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:07.051329 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.125:45305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 20:22:07.052668 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.052276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:45305 (local address 127.0.0.1:37088)
0929 20:22:07.052433 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:07.052521 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.125:45305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 20:22:07.052876 18428 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.125:45305: Network error: Client connection negotiation failed: client connection to 127.27.185.125:45305: connect: Connection refused (error 111)
I20260929 20:22:07.121172 18428 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.125" port: 45305 } attempt: 2
I20260929 20:22:07.122534 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.121948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:45305 (local address 127.0.0.1:37102)
0929 20:22:07.122240 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:07.122357 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.125:45305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 20:22:07.123729 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.123293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:45305 (local address 127.0.0.1:37118)
0929 20:22:07.123482 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:07.123579 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.125:45305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 20:22:07.123997 18428 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.125:45305: Network error: Client connection negotiation failed: client connection to 127.27.185.125:45305: connect: Connection refused (error 111)
I20260929 20:22:07.215274 18428 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.125" port: 45305 } attempt: 3
I20260929 20:22:07.216719 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.216064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:45305 (local address 127.0.0.1:37122)
0929 20:22:07.216347 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:07.216490 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.125:45305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 20:22:07.218035 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.217520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:45305 (local address 127.0.0.1:37130)
0929 20:22:07.217709 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:07.217812 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.125:45305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 20:22:07.218322 18428 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.125:45305: Network error: Client connection negotiation failed: client connection to 127.27.185.125:45305: connect: Connection refused (error 111)
W20260929 20:22:07.357229 18436 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 20:22:07.357697 18436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:07.357796 18436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:07.357920 18436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 20:22:07.361631 18428 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.125" port: 45305 } attempt: 4
I20260929 20:22:07.362942 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.362299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:45305 (local address 127.0.0.1:37140)
0929 20:22:07.362643 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:07.362764 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.125:45305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
I20260929 20:22:07.364111 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.363692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:45305 (local address 127.0.0.1:37142)
0929 20:22:07.363854 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:07.363952 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.125:45305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 20:22:07.364319 18428 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.125:45305: Network error: Client connection negotiation failed: client connection to 127.27.185.125:45305: connect: Connection refused (error 111)
W20260929 20:22:07.366272 18436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:22:07.366353 18436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:07.366410 18436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:07.366467 18436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:22:07.366534 18436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:22:07.376718 18436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:34175,127.27.185.125:45305,127.27.185.124:33429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.125.keytab
--principal=kudu/127.27.185.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.125:45305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.185.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:07.378052 18436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:07.379359 18436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:07.391484 18441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:07.391585 18444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:07.391619 18442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:07.392575 18436 server_base.cc:1034] running on GCE node
I20260929 20:22:07.393399 18436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:07.394850 18436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:07.396116 18436 hybrid_clock.cc:648] HybridClock initialized: now 1790713327396052 us; error 37 us; skew 500 ppm
I20260929 20:22:07.396469 18436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:07 dist-test-slave-4cx3 krb5kdc[18348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713327, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:07.399700 18436 init.cc:377] Logged in from keytab as kudu/127.27.185.125@KRBTEST.COM (short username kudu)
I20260929 20:22:07.401243 18436 webserver.cc:460] Webserver started at http://127.27.185.125:44161/ using document root <none> and password file <none>
I20260929 20:22:07.401966 18436 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:07.402101 18436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:07.402426 18436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:07.405493 18436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-1/data/instance:
uuid: "cfb3f0b9926d4307b9a8cf493557de06"
format_stamp: "Formatted at 2026-09-29 20:22:07 on dist-test-slave-4cx3"
server_key: "8306350dcce3200021692fb63c5b95fd"
server_key_iv: "bc7e7de5eab4f301ba369a696d6bbdf4"
server_key_version: "encryptionkey@0"
I20260929 20:22:07.406304 18436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-1/wal/instance:
uuid: "cfb3f0b9926d4307b9a8cf493557de06"
format_stamp: "Formatted at 2026-09-29 20:22:07 on dist-test-slave-4cx3"
server_key: "8306350dcce3200021692fb63c5b95fd"
server_key_iv: "bc7e7de5eab4f301ba369a696d6bbdf4"
server_key_version: "encryptionkey@0"
I20260929 20:22:07.411672 18436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 20:22:07.415081 18451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:07.416704 18436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:22:07.416882 18436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-1/wal
uuid: "cfb3f0b9926d4307b9a8cf493557de06"
format_stamp: "Formatted at 2026-09-29 20:22:07 on dist-test-slave-4cx3"
server_key: "8306350dcce3200021692fb63c5b95fd"
server_key_iv: "bc7e7de5eab4f301ba369a696d6bbdf4"
server_key_version: "encryptionkey@0"
I20260929 20:22:07.417117 18436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:07.453709 18436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:07.457878 18436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:07.458163 18436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:07.477607 18436 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.125:45305
I20260929 20:22:07.477639 18502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.125:45305 every 8 connection(s)
I20260929 20:22:07.479031 18436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-1/data/info.pb
I20260929 20:22:07.483068 18503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:07.488602 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 18436
I20260929 20:22:07.488790 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-1/wal/instance
I20260929 20:22:07.489349 28389 external_mini_cluster.cc:1428] Setting key a92c1f27e6c90a2a0b43059c1671bfd7
I20260929 20:22:07.490253 18503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34175 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:22:07 dist-test-slave-4cx3 krb5kdc[18348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713327, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.125@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:22:07.513417 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.493797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39738 (local address 127.27.185.126:34175)
0929 20:22:07.494034 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:07.494041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:07.495350 (+ 1309us) server_negotiation.cc:407] Connection header received
0929 20:22:07.497342 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:07.497351 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:07.497448 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:07.497569 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:07.499062 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:07.499830 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:07.502375 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:07.502796 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:07.502922 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:07.507037 (+ 4115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:07.507042 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:07.507046 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:07.507079 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:07.510321 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:07.511081 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:07.511088 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:07.511092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:07.511225 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:07.511682 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:07.511688 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:07.511692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:07.511979 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:07.512139 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:07.512849 (+ 710us) server_negotiation.cc:299] Negotiation successful
0929 20:22:07.513070 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 20:22:07.514418 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.494328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:34175 (local address 127.0.0.1:39738)
0929 20:22:07.495160 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:07.495213 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:07.497062 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:07.497772 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:07.497779 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:07.498299 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:07.498871 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:07.498888 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:07.500006 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:07.500014 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:07.502191 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:07.502209 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:07.503044 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:07.503053 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:07.503368 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:07.504131 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:07.504164 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:07.506788 (+ 2624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:07.510481 (+ 3693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:07.510492 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:07.510508 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:07.510950 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:07.511340 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:07.511345 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:07.511349 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:07.511568 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:07.512112 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:07.512120 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:07.512478 (+ 358us) client_negotiation.cc:769] Sending connection context
0929 20:22:07.512740 (+ 262us) client_negotiation.cc:240] Negotiation successful
0929 20:22:07.513283 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":372,"threads_started":1}
WARNING: no policy specified for kudu/127.27.185.124@KRBTEST.COM; defaulting to no policy
I20260929 20:22:07.516229 18503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.125" port: 45305 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.124@KRBTEST.COM" created.
Sep 29 20:22:07 dist-test-slave-4cx3 krb5kdc[18348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713327, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.125@KRBTEST.COM for kudu/127.27.185.125@KRBTEST.COM
I20260929 20:22:07.530869 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.517235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:45305 (local address 127.0.0.1:37154)
0929 20:22:07.517455 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:07.517473 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:07.517587 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:07.518442 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:07.518446 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:07.518797 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:07.518935 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:07.518943 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:07.520529 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:07.520537 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:07.521548 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:07.521559 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:07.522374 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:07.522379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:07.522554 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:07.523159 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:07.523176 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:07.525178 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:07.528369 (+ 3191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:07.528375 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:07.528379 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:07.528768 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:07.529167 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:07.529172 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:07.529175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:07.529295 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:07.529890 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:07.529897 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:07.530036 (+ 139us) client_negotiation.cc:769] Sending connection context
0929 20:22:07.530181 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 20:22:07.530474 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 20:22:07.530977 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.517306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37154 (local address 127.27.185.125:45305)
0929 20:22:07.518049 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:07.518055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:07.518082 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:22:07.518122 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:07.518128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:07.518299 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:07.518421 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:07.519120 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:07.520386 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:07.521672 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:07.522226 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:07.522366 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:07.525391 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:07.525403 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:07.525416 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:07.525447 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:07.528229 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:07.528918 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:07.528924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:07.528926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:07.529037 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:07.529423 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:07.529430 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:07.529434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:07.529764 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:07.529988 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:07.530328 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 20:22:07.530590 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":530,"thread_start_us":402,"threads_started":1}
I20260929 20:22:07.533545 18503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.124" port: 33429 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:22:07.534780 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.534304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:33429 (local address 127.0.0.1:55972)
0929 20:22:07.534500 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:07.534634 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:33429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 20:22:07.535087 18453 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.185.124:33429: connect: Connection refused (error 111)
I20260929 20:22:07.536882 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.536395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:33429 (local address 127.0.0.1:55976)
0929 20:22:07.536615 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:07.536726 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:33429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 20:22:07.537133 18503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.124:33429: Network error: Client connection negotiation failed: client connection to 127.27.185.124:33429: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.124.keytab.
Entry for principal kudu/127.27.185.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.124.keytab.
I20260929 20:22:07.551774 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.124:33429
--webserver_interface=127.27.185.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.124.keytab
--principal=kudu/127.27.185.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:34175,127.27.185.125:45305,127.27.185.124:33429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:22:07.586501 18503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.124" port: 33429 } attempt: 1
I20260929 20:22:07.587800 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.587232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:33429 (local address 127.0.0.1:55990)
0929 20:22:07.587482 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:07.587613 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:33429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 20:22:07.588936 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.588535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:33429 (local address 127.0.0.1:55996)
0929 20:22:07.588708 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:07.588797 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:33429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 20:22:07.589182 18503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.124:33429: Network error: Client connection negotiation failed: client connection to 127.27.185.124:33429: connect: Connection refused (error 111)
I20260929 20:22:07.657449 18503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.124" port: 33429 } attempt: 2
I20260929 20:22:07.658913 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.658341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:33429 (local address 127.0.0.1:56012)
0929 20:22:07.658600 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:07.658721 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:33429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 20:22:07.660048 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.659629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:33429 (local address 127.0.0.1:56014)
0929 20:22:07.659813 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:07.659900 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:33429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 20:22:07.660364 18503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.124:33429: Network error: Client connection negotiation failed: client connection to 127.27.185.124:33429: connect: Connection refused (error 111)
I20260929 20:22:07.663546 18428 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.125" port: 45305 } attempt: 5
Sep 29 20:22:07 dist-test-slave-4cx3 krb5kdc[18348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713326, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for kudu/127.27.185.125@KRBTEST.COM
I20260929 20:22:07.677474 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.664351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37166 (local address 127.27.185.125:45305)
0929 20:22:07.664536 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:07.664545 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:07.664566 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:22:07.664787 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:07.664792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:07.664863 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:07.664989 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:07.665811 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:07.666515 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:07.667731 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:07.668171 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:07.668308 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:07.671698 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:07.671703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:07.671707 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:07.671740 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:07.674664 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:07.675521 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:07.675525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:07.675528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:07.675656 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:07.676121 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:07.676124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:07.676126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:07.676409 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:07.676518 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:07.677042 (+ 524us) server_negotiation.cc:299] Negotiation successful
0929 20:22:07.677211 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 20:22:07.677795 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.664260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:45305 (local address 127.0.0.1:37166)
0929 20:22:07.664484 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:07.664505 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:07.664649 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:07.665028 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:07.665035 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:07.665474 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:07.665665 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:07.665675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:07.666654 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:07.666661 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:07.667581 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:07.667592 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:07.668332 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:07.668338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:07.668562 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:07.669203 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:07.669221 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:07.671486 (+ 2265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:07.674825 (+ 3339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:07.674830 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:07.674834 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:07.675360 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:07.675781 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:07.675788 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:07.675793 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:07.675999 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:07.676547 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:07.676553 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:07.676763 (+ 210us) client_negotiation.cc:769] Sending connection context
0929 20:22:07.676944 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 20:22:07.677303 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 20:22:07.678846 18428 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.124" port: 33429 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:22:07.679929 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.679540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:33429 (local address 127.0.0.1:56020)
0929 20:22:07.679699 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:07.679776 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:33429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 20:22:07.680980 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.680615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:33429 (local address 127.0.0.1:56030)
0929 20:22:07.680780 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:07.680878 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:33429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 20:22:07.681273 18428 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.124:33429: Network error: Client connection negotiation failed: client connection to 127.27.185.124:33429: connect: Connection refused (error 111)
I20260929 20:22:07.732569 18428 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.124" port: 33429 } attempt: 1
I20260929 20:22:07.734081 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.733442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:33429 (local address 127.0.0.1:56040)
0929 20:22:07.733684 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:07.733786 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:33429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 20:22:07.735453 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.735042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:33429 (local address 127.0.0.1:56044)
0929 20:22:07.735185 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:07.735269 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:33429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 20:22:07.735666 18428 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.124:33429: Network error: Client connection negotiation failed: client connection to 127.27.185.124:33429: connect: Connection refused (error 111)
I20260929 20:22:07.751636 18503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.124" port: 33429 } attempt: 3
I20260929 20:22:07.752923 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.752386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:33429 (local address 127.0.0.1:56054)
0929 20:22:07.752660 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:07.752750 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:33429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260929 20:22:07.753985 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.753563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:33429 (local address 127.0.0.1:56056)
0929 20:22:07.753790 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:07.753874 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:33429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260929 20:22:07.754171 18503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.124:33429: Network error: Client connection negotiation failed: client connection to 127.27.185.124:33429: connect: Connection refused (error 111)
I20260929 20:22:07.803889 18428 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.124" port: 33429 } attempt: 2
I20260929 20:22:07.805150 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.804659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:33429 (local address 127.0.0.1:56070)
0929 20:22:07.804894 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:07.804988 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:33429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 20:22:07.806201 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.805878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:33429 (local address 127.0.0.1:56076)
0929 20:22:07.806026 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:07.806097 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:33429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 20:22:07.806447 18428 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.124:33429: Network error: Client connection negotiation failed: client connection to 127.27.185.124:33429: connect: Connection refused (error 111)
W20260929 20:22:07.891008 18510 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 20:22:07.891489 18510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:07.891637 18510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:07.891786 18510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 20:22:07.897396 18503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.124" port: 33429 } attempt: 4
I20260929 20:22:07.898658 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.898175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:33429 (local address 127.0.0.1:56090)
0929 20:22:07.898409 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:07.898506 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:33429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 20:22:07.899667 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.899318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:33429 (local address 127.0.0.1:56096)
0929 20:22:07.899485 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:07.899562 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:33429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 20:22:07.899940 18510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:22:07.899916 18503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.124:33429: Network error: Client connection negotiation failed: client connection to 127.27.185.124:33429: connect: Connection refused (error 111)
W20260929 20:22:07.900046 18510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:07.900151 18510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:07.900242 18510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:22:07.900332 18510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:22:07.910751 18510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:34175,127.27.185.125:45305,127.27.185.124:33429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.124.keytab
--principal=kudu/127.27.185.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.124:33429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.185.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:07.912333 18510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:07.912648 18428 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.124" port: 33429 } attempt: 3
I20260929 20:22:07.913772 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.913259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:33429 (local address 127.0.0.1:56108)
0929 20:22:07.913532 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:07.913621 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:33429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 20:22:07.913810 18510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 20:22:07.914988 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:07.914584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:33429 (local address 127.0.0.1:56118)
0929 20:22:07.914800 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:07.914879 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:33429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260929 20:22:07.915257 18428 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.124:33429: Network error: Client connection negotiation failed: client connection to 127.27.185.124:33429: connect: Connection refused (error 111)
W20260929 20:22:07.926421 18515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:07.926374 18516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:07.926460 18518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:07.928049 18510 server_base.cc:1034] running on GCE node
I20260929 20:22:07.929097 18510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:07.931100 18510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:07.932436 18510 hybrid_clock.cc:648] HybridClock initialized: now 1790713327932266 us; error 63 us; skew 500 ppm
I20260929 20:22:07.932891 18510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:07 dist-test-slave-4cx3 krb5kdc[18348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713327, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:07.936251 18510 init.cc:377] Logged in from keytab as kudu/127.27.185.124@KRBTEST.COM (short username kudu)
I20260929 20:22:07.937978 18510 webserver.cc:460] Webserver started at http://127.27.185.124:43985/ using document root <none> and password file <none>
I20260929 20:22:07.938715 18510 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:07.938858 18510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:07.939189 18510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:07.942334 18510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-2/data/instance:
uuid: "76074d8087d842ad9d4df5bc7fa66fca"
format_stamp: "Formatted at 2026-09-29 20:22:07 on dist-test-slave-4cx3"
server_key: "6b241227fc9e6de865e341a9ebbbbcfb"
server_key_iv: "f7930fb63a34d8bde93a4e9036ba4139"
server_key_version: "encryptionkey@0"
I20260929 20:22:07.943202 18510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-2/wal/instance:
uuid: "76074d8087d842ad9d4df5bc7fa66fca"
format_stamp: "Formatted at 2026-09-29 20:22:07 on dist-test-slave-4cx3"
server_key: "6b241227fc9e6de865e341a9ebbbbcfb"
server_key_iv: "f7930fb63a34d8bde93a4e9036ba4139"
server_key_version: "encryptionkey@0"
I20260929 20:22:07.948621 18510 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 20:22:07.952472 18525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:07.953998 18510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:22:07.954193 18510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-2/wal
uuid: "76074d8087d842ad9d4df5bc7fa66fca"
format_stamp: "Formatted at 2026-09-29 20:22:07 on dist-test-slave-4cx3"
server_key: "6b241227fc9e6de865e341a9ebbbbcfb"
server_key_iv: "f7930fb63a34d8bde93a4e9036ba4139"
server_key_version: "encryptionkey@0"
I20260929 20:22:07.954422 18510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:07.969792 18510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:07.974067 18510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:07.974413 18510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:07.993659 18510 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.124:33429
I20260929 20:22:07.993680 18577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.124:33429 every 8 connection(s)
I20260929 20:22:07.995062 18510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-2/data/info.pb
I20260929 20:22:07.997326 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 18510
I20260929 20:22:07.997534 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790712966917858-28389-0/minicluster-data/master-2/wal/instance
I20260929 20:22:07.998013 28389 external_mini_cluster.cc:1428] Setting key 410e380dd6b447c24fc96b83c19196d1
I20260929 20:22:07.999279 18578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:08.009846 18578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34175 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:22:08 dist-test-slave-4cx3 krb5kdc[18348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.124@KRBTEST.COM
Sep 29 20:22:08 dist-test-slave-4cx3 krb5kdc[18348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713327, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.124@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:22:08.023026 18581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:08.000089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56128 (local address 127.27.185.124:33429)
0929 20:22:08.001399 (+ 1310us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:08.001416 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:08.001474 (+ 58us) server_negotiation.cc:407] Connection header received
0929 20:22:08.003420 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:08.003451 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:08.004014 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:08.004492 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:08.005805 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.007889 (+ 2084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.009121 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.010186 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.010395 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:08.015217 (+ 4822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:08.015225 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:08.015234 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:08.015272 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:08.018587 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:08.019244 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:08.019253 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:08.019261 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:08.019512 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:08.019896 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:08.019901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:08.019905 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:08.020522 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:08.020837 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:08.021298 (+ 461us) server_negotiation.cc:299] Negotiation successful
0929 20:22:08.021757 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1046,"thread_start_us":759,"threads_started":1}
I20260929 20:22:08.028640 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:08.012838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39754 (local address 127.27.185.126:34175)
0929 20:22:08.013070 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:08.013074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:08.013892 (+ 818us) server_negotiation.cc:407] Connection header received
0929 20:22:08.014086 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:08.014093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:08.014169 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:08.014250 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:08.015756 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.016898 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.018421 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.018854 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.019006 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:08.022322 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:08.022329 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:08.022333 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:08.022373 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:08.025645 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:08.026505 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:08.026515 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:08.026518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:08.026715 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:08.027080 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:08.027089 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:08.027093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:08.027353 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:08.027569 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:08.028004 (+ 435us) server_negotiation.cc:299] Negotiation successful
0929 20:22:08.028217 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260929 20:22:08.028656 18582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:08.012866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:34175 (local address 127.0.0.1:39754)
0929 20:22:08.013721 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:08.013772 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:08.013943 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:08.014348 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:08.014358 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:08.015232 (+ 874us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:08.015606 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:08.015617 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.017077 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.017083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:08.018224 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:08.018237 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.019038 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.019044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:08.019222 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:08.019840 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:08.019856 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:08.022130 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:08.025771 (+ 3641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:08.025783 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:08.025790 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:08.026396 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:08.026822 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:08.026826 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:08.026828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:08.026963 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:08.027537 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:08.027545 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:08.027727 (+ 182us) client_negotiation.cc:769] Sending connection context
0929 20:22:08.027910 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 20:22:08.028216 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":415,"threads_started":1}
I20260929 20:22:08.030030 18578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.125" port: 45305 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:22:08 dist-test-slave-4cx3 krb5kdc[18348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
Sep 29 20:22:08 dist-test-slave-4cx3 krb5kdc[18348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713327, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.124@KRBTEST.COM for kudu/127.27.185.125@KRBTEST.COM
I20260929 20:22:08.042325 18583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:08.026091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39770 (local address 127.27.185.126:34175)
0929 20:22:08.026712 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:08.026717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:08.026740 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:22:08.026785 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:08.026793 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:08.026859 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:08.026976 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:08.028241 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.029349 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.030640 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.031252 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.031366 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:08.036238 (+ 4872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:08.036243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:08.036246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:08.036276 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:08.039656 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:08.040341 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:08.040346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:08.040350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:08.040512 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:08.040881 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:08.040886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:08.040889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:08.041244 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:08.041515 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:08.041699 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 20:22:08.041932 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":240,"threads_started":1}
I20260929 20:22:08.044809 18582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:08.030667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:45305 (local address 127.0.0.1:37170)
0929 20:22:08.030840 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:08.030858 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:08.030944 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:08.031278 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:08.031281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:08.031805 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:08.031991 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:08.032000 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.032985 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.032992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:08.034502 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:08.034514 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.035524 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.035532 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:08.035726 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:08.036275 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:08.036291 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:08.038306 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:08.041986 (+ 3680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:08.041991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:08.041994 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:08.042436 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:08.042921 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:08.042928 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:08.042932 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:08.043111 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:08.043802 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:08.043806 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:08.044003 (+ 197us) client_negotiation.cc:769] Sending connection context
0929 20:22:08.044207 (+ 204us) client_negotiation.cc:240] Negotiation successful
0929 20:22:08.044483 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 20:22:08.045329 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:08.030799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37170 (local address 127.27.185.125:45305)
0929 20:22:08.030982 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:08.030990 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:08.031009 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:22:08.031068 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:08.031075 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:08.031152 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:08.031257 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:08.032173 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.032834 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.034636 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.035302 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.035465 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:08.038502 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:08.038511 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:08.038517 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:08.038562 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:08.041836 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:08.042599 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:08.042605 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:08.042608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:08.042774 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:08.043221 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:08.043225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:08.043227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:08.043677 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:08.043828 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:08.044297 (+ 469us) server_negotiation.cc:299] Negotiation successful
0929 20:22:08.044590 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 20:22:08.046330 18578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.124" port: 33429 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:22:08 dist-test-slave-4cx3 krb5kdc[18348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713327, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.124@KRBTEST.COM for kudu/127.27.185.124@KRBTEST.COM
Sep 29 20:22:08 dist-test-slave-4cx3 krb5kdc[18348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.125@KRBTEST.COM
I20260929 20:22:08.060286 18582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:08.047131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:33429 (local address 127.0.0.1:56144)
0929 20:22:08.047315 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:08.047329 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:08.047463 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:08.047778 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:08.047782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:08.048261 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:08.048424 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:08.048435 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.049656 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.049660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:08.050736 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:08.050750 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.051684 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.051691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:08.051936 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:08.052444 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:08.052461 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:08.054539 (+ 2078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:08.057512 (+ 2973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:08.057517 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:08.057521 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:08.057991 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:08.058476 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:08.058481 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:08.058484 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:08.058654 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:08.059300 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:08.059304 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:08.059476 (+ 172us) client_negotiation.cc:769] Sending connection context
0929 20:22:08.059615 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 20:22:08.059870 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260929 20:22:08.060380 18581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:08.047074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56144 (local address 127.27.185.124:33429)
0929 20:22:08.047249 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:08.047256 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:08.047415 (+ 159us) server_negotiation.cc:407] Connection header received
0929 20:22:08.047574 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:08.047579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:08.047647 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:08.047725 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:08.048558 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.049532 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.050875 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.051522 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.051633 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:08.054722 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:08.054731 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:08.054739 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:08.054780 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:08.057389 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:08.058107 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:08.058110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:08.058112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:08.058228 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:08.058765 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:08.058768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:08.058770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:08.059179 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:08.059326 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:08.059747 (+ 421us) server_negotiation.cc:299] Negotiation successful
0929 20:22:08.060011 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":62}
I20260929 20:22:08.061860 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:08.044216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37178 (local address 127.27.185.125:45305)
0929 20:22:08.045642 (+ 1426us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:08.045654 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:08.045676 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:22:08.045741 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:08.045748 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:08.045837 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:08.045957 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:08.047619 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.048705 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.050232 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.050912 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.050999 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:08.055551 (+ 4552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:08.055560 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:08.055568 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:08.055609 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:08.058943 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:08.059833 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:08.059838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:08.059840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:08.060047 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:08.060500 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:08.060505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:08.060509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:08.060791 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:08.061080 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:08.061313 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 20:22:08.061527 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1287,"thread_start_us":1717,"threads_started":1}
I20260929 20:22:08.066135 18578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76074d8087d842ad9d4df5bc7fa66fca: Bootstrap starting.
I20260929 20:22:08.066404 28389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:22:08.069252 18578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76074d8087d842ad9d4df5bc7fa66fca: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:08.070312 18578 log.cc:824] T 00000000000000000000000000000000 P 76074d8087d842ad9d4df5bc7fa66fca: Log is configured to *not* fsync() on all Append() calls
I20260929 20:22:08.069007 18403 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "07eb795753c540b082eb53db45b9c522"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39770
I20260929 20:22:08.073897 18578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76074d8087d842ad9d4df5bc7fa66fca: No bootstrap required, opened a new log
I20260929 20:22:08.085638 18578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76074d8087d842ad9d4df5bc7fa66fca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eb795753c540b082eb53db45b9c522" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34175 } } peers { permanent_uuid: "cfb3f0b9926d4307b9a8cf493557de06" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 45305 } } peers { permanent_uuid: "76074d8087d842ad9d4df5bc7fa66fca" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 33429 } }
I20260929 20:22:08.086112 18578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76074d8087d842ad9d4df5bc7fa66fca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76074d8087d842ad9d4df5bc7fa66fca, State: Initialized, Role: FOLLOWER
I20260929 20:22:08.086663 18578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76074d8087d842ad9d4df5bc7fa66fca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eb795753c540b082eb53db45b9c522" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34175 } } peers { permanent_uuid: "cfb3f0b9926d4307b9a8cf493557de06" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 45305 } } peers { permanent_uuid: "76074d8087d842ad9d4df5bc7fa66fca" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 33429 } }
I20260929 20:22:08.087682 18586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76074d8087d842ad9d4df5bc7fa66fca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eb795753c540b082eb53db45b9c522" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34175 } } peers { permanent_uuid: "cfb3f0b9926d4307b9a8cf493557de06" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 45305 } } peers { permanent_uuid: "76074d8087d842ad9d4df5bc7fa66fca" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 33429 } } }
I20260929 20:22:08.088038 18586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76074d8087d842ad9d4df5bc7fa66fca [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:22:08.089054 18578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76074d8087d842ad9d4df5bc7fa66fca [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:22:08.092499 18428 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.124" port: 33429 } attempt: 4
W20260929 20:22:08.101033 18597 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 76074d8087d842ad9d4df5bc7fa66fca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:22:08.101190 18597 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 29 20:22:08 dist-test-slave-4cx3 krb5kdc[18348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713326, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for kudu/127.27.185.124@KRBTEST.COM
I20260929 20:22:08.097410 18510 master_runner.cc:186] Error getting master registration for 127.27.185.126:34175: OK, instance_id { permanent_uuid: "07eb795753c540b082eb53db45b9c522" instance_seqno: 1790713326947295 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 20:22:08.109201 18510 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 20:22:08.111202 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:08.093239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:33429 (local address 127.0.0.1:56148)
0929 20:22:08.093510 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:08.093532 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:08.093672 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:08.094036 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:08.094041 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:08.094445 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:08.094723 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:08.094738 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.097048 (+ 2310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.097055 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:08.098754 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:08.098770 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.100220 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.100229 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:08.100433 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:08.101413 (+ 980us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:08.101439 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:08.104272 (+ 2833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:08.108192 (+ 3920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:08.108198 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:08.108202 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:08.108709 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:08.109178 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:08.109184 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:08.109190 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:08.109400 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:08.110152 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:08.110157 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:08.110324 (+ 167us) client_negotiation.cc:769] Sending connection context
0929 20:22:08.110494 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 20:22:08.110795 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260929 20:22:08.111227 18581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:08.093295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56148 (local address 127.27.185.124:33429)
0929 20:22:08.093539 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:08.093550 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:08.093596 (+ 46us) server_negotiation.cc:407] Connection header received
0929 20:22:08.093817 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:08.093822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:08.093902 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:08.094003 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:08.094937 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.096086 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.098943 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.099821 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.099948 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:08.104477 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:08.104489 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:08.104493 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:08.104527 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:08.108012 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:08.108855 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:08.108861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:08.108864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:08.109029 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:08.109533 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:08.109539 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:08.109542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:08.110008 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:08.110180 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:08.110618 (+ 438us) server_negotiation.cc:299] Negotiation successful
0929 20:22:08.110861 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 20:22:08.117487 18428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07eb795753c540b082eb53db45b9c522: Bootstrap starting.
I20260929 20:22:08.121763 18428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07eb795753c540b082eb53db45b9c522: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:08.123157 18428 log.cc:824] T 00000000000000000000000000000000 P 07eb795753c540b082eb53db45b9c522: Log is configured to *not* fsync() on all Append() calls
I20260929 20:22:08.126719 18428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07eb795753c540b082eb53db45b9c522: No bootstrap required, opened a new log
I20260929 20:22:08.134514 18428 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07eb795753c540b082eb53db45b9c522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eb795753c540b082eb53db45b9c522" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34175 } } peers { permanent_uuid: "cfb3f0b9926d4307b9a8cf493557de06" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 45305 } } peers { permanent_uuid: "76074d8087d842ad9d4df5bc7fa66fca" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 33429 } }
I20260929 20:22:08.134979 18428 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07eb795753c540b082eb53db45b9c522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07eb795753c540b082eb53db45b9c522, State: Initialized, Role: FOLLOWER
I20260929 20:22:08.135562 18428 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07eb795753c540b082eb53db45b9c522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eb795753c540b082eb53db45b9c522" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34175 } } peers { permanent_uuid: "cfb3f0b9926d4307b9a8cf493557de06" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 45305 } } peers { permanent_uuid: "76074d8087d842ad9d4df5bc7fa66fca" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 33429 } }
I20260929 20:22:08.136580 18600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07eb795753c540b082eb53db45b9c522 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eb795753c540b082eb53db45b9c522" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34175 } } peers { permanent_uuid: "cfb3f0b9926d4307b9a8cf493557de06" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 45305 } } peers { permanent_uuid: "76074d8087d842ad9d4df5bc7fa66fca" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 33429 } } }
I20260929 20:22:08.136991 18600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07eb795753c540b082eb53db45b9c522 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:22:08.137965 18428 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07eb795753c540b082eb53db45b9c522 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:22:08.138007 18403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07eb795753c540b082eb53db45b9c522 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 20:22:08.138247 18403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07eb795753c540b082eb53db45b9c522 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:08.140779 18403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07eb795753c540b082eb53db45b9c522 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eb795753c540b082eb53db45b9c522" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34175 } } peers { permanent_uuid: "cfb3f0b9926d4307b9a8cf493557de06" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 45305 } } peers { permanent_uuid: "76074d8087d842ad9d4df5bc7fa66fca" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 33429 } }
I20260929 20:22:08.142536 18403 leader_election.cc:288] T 00000000000000000000000000000000 P 07eb795753c540b082eb53db45b9c522 [CANDIDATE]: Term 1 election: Requested vote from peers cfb3f0b9926d4307b9a8cf493557de06 (127.27.185.125:45305), 76074d8087d842ad9d4df5bc7fa66fca (127.27.185.124:33429)
I20260929 20:22:08.143342 18552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07eb795753c540b082eb53db45b9c522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "76074d8087d842ad9d4df5bc7fa66fca"
I20260929 20:22:08.143797 18552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76074d8087d842ad9d4df5bc7fa66fca [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:08.146119 18552 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 76074d8087d842ad9d4df5bc7fa66fca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07eb795753c540b082eb53db45b9c522 in term 1.
I20260929 20:22:08.146714 18378 leader_election.cc:302] T 00000000000000000000000000000000 P 07eb795753c540b082eb53db45b9c522 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07eb795753c540b082eb53db45b9c522, 76074d8087d842ad9d4df5bc7fa66fca; no voters:
I20260929 20:22:08.147185 18600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07eb795753c540b082eb53db45b9c522 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:22:08.185005 18600 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07eb795753c540b082eb53db45b9c522 [term 1 LEADER]: Becoming Leader. State: Replica: 07eb795753c540b082eb53db45b9c522, State: Running, Role: LEADER
I20260929 20:22:08.185755 18600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07eb795753c540b082eb53db45b9c522 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eb795753c540b082eb53db45b9c522" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34175 } } peers { permanent_uuid: "cfb3f0b9926d4307b9a8cf493557de06" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 45305 } } peers { permanent_uuid: "76074d8087d842ad9d4df5bc7fa66fca" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 33429 } }
W20260929 20:22:08.187003 18382 tablet.cc:2369] T 00000000000000000000000000000000 P 07eb795753c540b082eb53db45b9c522: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:22:08.199128 18503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.124" port: 33429 } attempt: 5
I20260929 20:22:08.201099 18604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07eb795753c540b082eb53db45b9c522 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07eb795753c540b082eb53db45b9c522. Latest consensus state: current_term: 1 leader_uuid: "07eb795753c540b082eb53db45b9c522" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eb795753c540b082eb53db45b9c522" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34175 } } peers { permanent_uuid: "cfb3f0b9926d4307b9a8cf493557de06" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 45305 } } peers { permanent_uuid: "76074d8087d842ad9d4df5bc7fa66fca" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 33429 } } }
I20260929 20:22:08.201586 18604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07eb795753c540b082eb53db45b9c522 [sys.catalog]: This master's current role is: LEADER
I20260929 20:22:08.202385 18617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 29 20:22:08 dist-test-slave-4cx3 krb5kdc[18348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713327, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.125@KRBTEST.COM for kudu/127.27.185.124@KRBTEST.COM
I20260929 20:22:08.219708 18617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:22:08.225435 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:08.199767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:33429 (local address 127.0.0.1:56164)
0929 20:22:08.199975 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:08.199995 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:08.200103 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:08.201954 (+ 1851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:08.201959 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:08.202528 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:08.202779 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:08.202795 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.206275 (+ 3480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.206281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:08.207722 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:08.207734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.209793 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.209800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:08.210097 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:08.211631 (+ 1534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:08.211657 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:08.215115 (+ 3458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:08.221655 (+ 6540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:08.221662 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:08.221667 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:08.222193 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:08.222990 (+ 797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:08.222996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:08.223000 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:08.223207 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:08.224354 (+ 1147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:08.224360 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:08.224555 (+ 195us) client_negotiation.cc:769] Sending connection context
0929 20:22:08.224696 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 20:22:08.225016 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
W20260929 20:22:08.226555 18531 tablet.cc:2369] T 00000000000000000000000000000000 P 76074d8087d842ad9d4df5bc7fa66fca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:22:08.227490 18616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:08.200048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56164 (local address 127.27.185.124:33429)
0929 20:22:08.201101 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:08.201108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:08.201159 (+ 51us) server_negotiation.cc:407] Connection header received
0929 20:22:08.201254 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:08.201293 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:08.201443 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:08.203081 (+ 1638us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:08.203701 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.206042 (+ 2341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.208005 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.209526 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.210601 (+ 1075us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:08.215496 (+ 4895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:08.215506 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:08.215511 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:08.215590 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:08.221445 (+ 5855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:08.222451 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:08.222458 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:08.222462 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:08.222801 (+ 339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:08.223440 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:08.223445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:08.223448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:08.224122 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:08.226101 (+ 1979us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:08.226338 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 20:22:08.226727 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":779,"thread_start_us":429,"threads_started":1}
I20260929 20:22:08.233682 18503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfb3f0b9926d4307b9a8cf493557de06: Bootstrap starting.
I20260929 20:22:08.237980 18503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfb3f0b9926d4307b9a8cf493557de06: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:08.239889 18503 log.cc:824] T 00000000000000000000000000000000 P cfb3f0b9926d4307b9a8cf493557de06: Log is configured to *not* fsync() on all Append() calls
I20260929 20:22:08.243127 18552 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 76074d8087d842ad9d4df5bc7fa66fca [term 1 FOLLOWER]: Refusing update from remote peer 07eb795753c540b082eb53db45b9c522: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:22:08.244396 18600 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 07eb795753c540b082eb53db45b9c522 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76074d8087d842ad9d4df5bc7fa66fca" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 33429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:22:08.248047 18624 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 20:22:08.253885 18503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfb3f0b9926d4307b9a8cf493557de06: No bootstrap required, opened a new log
I20260929 20:22:08.268647 18581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:08.174292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56152 (local address 127.27.185.124:33429)
0929 20:22:08.174517 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:08.174525 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:08.202053 (+ 27528us) server_negotiation.cc:407] Connection header received
0929 20:22:08.204341 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:08.204347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:08.204430 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:08.204510 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:08.213207 (+ 8697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.214345 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.222634 (+ 8289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.223330 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.223453 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:08.236560 (+ 13107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:08.236566 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:08.236571 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:08.236610 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:08.240657 (+ 4047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:08.263057 (+ 22400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:08.263065 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:08.263069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:08.263283 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:08.264830 (+ 1547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:08.264836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:08.264839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:08.265222 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:08.265524 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:08.267905 (+ 2381us) server_negotiation.cc:299] Negotiation successful
0929 20:22:08.268207 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"spinlock_wait_cycles":24448}
I20260929 20:22:08.255889 18361 master_runner.cc:186] Error getting master registration for 127.27.185.125:45305: OK, instance_id { permanent_uuid: "cfb3f0b9926d4307b9a8cf493557de06" instance_seqno: 1790713327472041 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 20:22:08.269271 18361 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 20:22:08.270241 18583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:08.170366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39776 (local address 127.27.185.126:34175)
0929 20:22:08.170631 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:08.170641 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:08.196340 (+ 25699us) server_negotiation.cc:407] Connection header received
0929 20:22:08.197897 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:08.197903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:08.198104 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:08.198201 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:08.210080 (+ 11879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.216141 (+ 6061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.220675 (+ 4534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.221468 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.221583 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:08.244513 (+ 22930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:08.244520 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:08.244524 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:08.244565 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:08.261377 (+ 16812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:08.263893 (+ 2516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:08.263900 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:08.263906 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:08.264078 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:08.266162 (+ 2084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:08.266169 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:08.266174 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:08.266569 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:08.266821 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:08.269619 (+ 2798us) server_negotiation.cc:299] Negotiation successful
0929 20:22:08.269866 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 20:22:08.272094 18503 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfb3f0b9926d4307b9a8cf493557de06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eb795753c540b082eb53db45b9c522" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34175 } } peers { permanent_uuid: "cfb3f0b9926d4307b9a8cf493557de06" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 45305 } } peers { permanent_uuid: "76074d8087d842ad9d4df5bc7fa66fca" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 33429 } }
I20260929 20:22:08.272784 18503 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfb3f0b9926d4307b9a8cf493557de06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfb3f0b9926d4307b9a8cf493557de06, State: Initialized, Role: FOLLOWER
I20260929 20:22:08.273769 18503 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfb3f0b9926d4307b9a8cf493557de06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eb795753c540b082eb53db45b9c522" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34175 } } peers { permanent_uuid: "cfb3f0b9926d4307b9a8cf493557de06" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 45305 } } peers { permanent_uuid: "76074d8087d842ad9d4df5bc7fa66fca" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 33429 } }
I20260929 20:22:08.289904 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:08.187522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37194 (local address 127.27.185.125:45305)
0929 20:22:08.188022 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:08.188031 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:08.207163 (+ 19132us) server_negotiation.cc:407] Connection header received
0929 20:22:08.207412 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:08.207417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:08.207492 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:08.207561 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:08.224470 (+ 16909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.225608 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.232693 (+ 7085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:08.233373 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:08.233475 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:08.275744 (+ 42269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:08.275754 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:08.275762 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:08.275822 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:08.280542 (+ 4720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:08.282970 (+ 2428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:08.282977 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:08.282982 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:08.283163 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:08.287271 (+ 4108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:08.287281 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:08.287287 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:08.287802 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:08.288119 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:08.289158 (+ 1039us) server_negotiation.cc:299] Negotiation successful
0929 20:22:08.289452 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356}
I20260929 20:22:08.299225 18586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76074d8087d842ad9d4df5bc7fa66fca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07eb795753c540b082eb53db45b9c522. Latest consensus state: current_term: 1 leader_uuid: "07eb795753c540b082eb53db45b9c522" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eb795753c540b082eb53db45b9c522" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34175 } } peers { permanent_uuid: "cfb3f0b9926d4307b9a8cf493557de06" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 45305 } } peers { permanent_uuid: "76074d8087d842ad9d4df5bc7fa66fca" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 33429 } } }
I20260929 20:22:08.299642 18586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76074d8087d842ad9d4df5bc7fa66fca [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:22:08.304644 18586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76074d8087d842ad9d4df5bc7fa66fca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "07eb795753c540b082eb53db45b9c522" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eb795753c540b082eb53db45b9c522" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34175 } } peers { permanent_uuid: "cfb3f0b9926d4307b9a8cf493557de06" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 45305 } } peers { permanent_uuid: "76074d8087d842ad9d4df5bc7fa66fca" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 33429 } } }
I20260929 20:22:08.304905 18586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76074d8087d842ad9d4df5bc7fa66fca [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:22:08.313422 18604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07eb795753c540b082eb53db45b9c522 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "07eb795753c540b082eb53db45b9c522" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eb795753c540b082eb53db45b9c522" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34175 } } peers { permanent_uuid: "cfb3f0b9926d4307b9a8cf493557de06" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 45305 } } peers { permanent_uuid: "76074d8087d842ad9d4df5bc7fa66fca" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 33429 } } }
I20260929 20:22:08.313707 18604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07eb795753c540b082eb53db45b9c522 [sys.catalog]: This master's current role is: LEADER
I20260929 20:22:08.314038 18617 catalog_manager.cc:1347] Generated new cluster ID: 393dc759429d48e09cf591cec7ca2b1b
I20260929 20:22:08.314141 18617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:22:08.314177 18621 mvcc.cc:205] Tried to move back new op lower bound from 7334761792460664832 to 7334761792261775360. Current Snapshot: MvccSnapshot[applied={T|T < 7334761792460664832 or (T in {7334761792460664832})}]
I20260929 20:22:08.343278 18503 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfb3f0b9926d4307b9a8cf493557de06 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:22:08.344929 18477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfb3f0b9926d4307b9a8cf493557de06 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:08.344914 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07eb795753c540b082eb53db45b9c522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cfb3f0b9926d4307b9a8cf493557de06"
I20260929 20:22:08.345790 18627 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfb3f0b9926d4307b9a8cf493557de06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eb795753c540b082eb53db45b9c522" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34175 } } peers { permanent_uuid: "cfb3f0b9926d4307b9a8cf493557de06" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 45305 } } peers { permanent_uuid: "76074d8087d842ad9d4df5bc7fa66fca" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 33429 } } }
I20260929 20:22:08.349012 18477 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cfb3f0b9926d4307b9a8cf493557de06 [term 1 FOLLOWER]: Refusing update from remote peer 07eb795753c540b082eb53db45b9c522: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:22:08.352126 18478 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cfb3f0b9926d4307b9a8cf493557de06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07eb795753c540b082eb53db45b9c522 in term 1.
I20260929 20:22:08.358385 18600 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 07eb795753c540b082eb53db45b9c522 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfb3f0b9926d4307b9a8cf493557de06" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 45305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:22:08.361876 18627 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfb3f0b9926d4307b9a8cf493557de06 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:22:08.385012 18617 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:22:08.386425 18626 mvcc.cc:205] Tried to move back new op lower bound from 7334761792460664832 to 7334761792261775360. Current Snapshot: MvccSnapshot[applied={T|T < 7334761792460664832}]
I20260929 20:22:08.386934 18617 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:22:08.406399 18627 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfb3f0b9926d4307b9a8cf493557de06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07eb795753c540b082eb53db45b9c522. Latest consensus state: current_term: 1 leader_uuid: "07eb795753c540b082eb53db45b9c522" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eb795753c540b082eb53db45b9c522" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34175 } } peers { permanent_uuid: "cfb3f0b9926d4307b9a8cf493557de06" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 45305 } } peers { permanent_uuid: "76074d8087d842ad9d4df5bc7fa66fca" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 33429 } } }
I20260929 20:22:08.406456 18632 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfb3f0b9926d4307b9a8cf493557de06 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "07eb795753c540b082eb53db45b9c522" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eb795753c540b082eb53db45b9c522" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34175 } } peers { permanent_uuid: "cfb3f0b9926d4307b9a8cf493557de06" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 45305 } } peers { permanent_uuid: "76074d8087d842ad9d4df5bc7fa66fca" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 33429 } } }
I20260929 20:22:08.406880 18627 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfb3f0b9926d4307b9a8cf493557de06 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:22:08.406913 18632 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfb3f0b9926d4307b9a8cf493557de06 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:22:08.413422 18617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07eb795753c540b082eb53db45b9c522: Generated new TSK 0
I20260929 20:22:08.416836 18617 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:22:08.428139 18600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07eb795753c540b082eb53db45b9c522 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "07eb795753c540b082eb53db45b9c522" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eb795753c540b082eb53db45b9c522" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 34175 } } peers { permanent_uuid: "cfb3f0b9926d4307b9a8cf493557de06" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 45305 } } peers { permanent_uuid: "76074d8087d842ad9d4df5bc7fa66fca" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 33429 } } }
I20260929 20:22:08.428640 18600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07eb795753c540b082eb53db45b9c522 [sys.catalog]: This master's current role is: LEADER
I20260929 20:22:08.485488 18649 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260929 20:22:08.542178 18583 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:22:08.532803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39792 (local address 127.27.185.126:34175)
0929 20:22:08.533050 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:08.533060 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:08.540006 (+ 6946us) server_negotiation.cc:407] Connection header received
0929 20:22:08.540315 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:08.540321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:08.540436 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:08.540518 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:08.541855 (+ 1337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39792: BlockingRecv error: recv got EOF from 127.0.0.1:39792 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260929 20:22:08.544489 18581 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:22:08.532965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56178 (local address 127.27.185.124:33429)
0929 20:22:08.533259 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:08.533268 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:08.542807 (+ 9539us) server_negotiation.cc:407] Connection header received
0929 20:22:08.543069 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:08.543076 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:08.543154 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:08.543225 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:08.544195 (+ 970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56178: BlockingRecv error: recv got EOF from 127.0.0.1:56178 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
W20260929 20:22:08.546298 18507 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:22:08.538886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37210 (local address 127.27.185.125:45305)
0929 20:22:08.539103 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:08.539112 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:08.542917 (+ 3805us) server_negotiation.cc:407] Connection header received
0929 20:22:08.544981 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:08.544987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:08.545072 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:08.545146 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:08.546062 (+ 916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37210: BlockingRecv error: recv got EOF from 127.0.0.1:37210 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260929 20:22:08.554955 18507 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:22:08.552780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37218 (local address 127.27.185.125:45305)
0929 20:22:08.552964 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:08.552970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:08.553450 (+ 480us) server_negotiation.cc:407] Connection header received
0929 20:22:08.553674 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:08.553682 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:08.553804 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:08.553920 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:08.554713 (+ 793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37218: BlockingRecv error: recv got EOF from 127.0.0.1:37218 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 20:22:08.564150 18507 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:22:08.559542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37224 (local address 127.27.185.125:45305)
0929 20:22:08.559810 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:08.559816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:08.562513 (+ 2697us) server_negotiation.cc:407] Connection header received
0929 20:22:08.562724 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:08.562730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:08.562820 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:08.562925 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:08.563928 (+ 1003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37224: BlockingRecv error: recv got EOF from 127.0.0.1:37224 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260929 20:22:08.573490 18507 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:22:08.569625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37236 (local address 127.27.185.125:45305)
0929 20:22:08.569813 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:08.569818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:08.571966 (+ 2148us) server_negotiation.cc:407] Connection header received
0929 20:22:08.572185 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:08.572191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:08.572284 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:08.572401 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:08.573248 (+ 847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37236: BlockingRecv error: recv got EOF from 127.0.0.1:37236 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260929 20:22:08.583366 18507 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:22:08.580996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37238 (local address 127.27.185.125:45305)
0929 20:22:08.581201 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:08.581207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:08.581770 (+ 563us) server_negotiation.cc:407] Connection header received
0929 20:22:08.582007 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:08.582013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:08.582104 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:08.582218 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:08.583117 (+ 899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37238: BlockingRecv error: recv got EOF from 127.0.0.1:37238 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260929 20:22:08.597468 18507 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:22:08.594910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37246 (local address 127.27.185.125:45305)
0929 20:22:08.595117 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:08.595124 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:08.595944 (+ 820us) server_negotiation.cc:407] Connection header received
0929 20:22:08.596160 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:08.596166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:08.596258 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:08.596372 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:08.597224 (+ 852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37246: BlockingRecv error: recv got EOF from 127.0.0.1:37246 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 20:22:08.619870 18507 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:22:08.617523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37252 (local address 127.27.185.125:45305)
0929 20:22:08.617715 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:08.617721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:08.618279 (+ 558us) server_negotiation.cc:407] Connection header received
0929 20:22:08.618478 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:08.618486 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:08.618599 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:08.618720 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:08.619611 (+ 891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37252: BlockingRecv error: recv got EOF from 127.0.0.1:37252 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 20:22:08.657869 18507 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:22:08.655553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37268 (local address 127.27.185.125:45305)
0929 20:22:08.655735 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:08.655742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:08.656263 (+ 521us) server_negotiation.cc:407] Connection header received
0929 20:22:08.656452 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:08.656458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:08.656564 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:08.656691 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:08.657606 (+ 915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37268: BlockingRecv error: recv got EOF from 127.0.0.1:37268 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 20:22:08.730002 18507 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:22:08.728029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37278 (local address 127.27.185.125:45305)
0929 20:22:08.728290 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:08.728298 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:08.728639 (+ 341us) server_negotiation.cc:407] Connection header received
0929 20:22:08.728807 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:08.728813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:08.728909 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:08.729028 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:08.729777 (+ 749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37278: BlockingRecv error: recv got EOF from 127.0.0.1:37278 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
W20260929 20:22:08.863421 18507 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:22:08.861398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37282 (local address 127.27.185.125:45305)
0929 20:22:08.861669 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:08.861677 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:08.861959 (+ 282us) server_negotiation.cc:407] Connection header received
0929 20:22:08.862200 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:08.862204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:08.862292 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:08.862410 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:08.863130 (+ 720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37282: BlockingRecv error: recv got EOF from 127.0.0.1:37282 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 20:22:09.103138 18597 catalog_manager.cc:1259] Loaded cluster ID: 393dc759429d48e09cf591cec7ca2b1b
I20260929 20:22:09.103252 18597 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 76074d8087d842ad9d4df5bc7fa66fca: loading cluster ID for follower catalog manager: success
I20260929 20:22:09.105774 18597 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 76074d8087d842ad9d4df5bc7fa66fca: acquiring CA information for follower catalog manager: success
I20260929 20:22:09.107630 18597 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 76074d8087d842ad9d4df5bc7fa66fca: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 20:22:09.125088 18507 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:22:09.123172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37288 (local address 127.27.185.125:45305)
0929 20:22:09.123413 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:09.123419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:09.123678 (+ 259us) server_negotiation.cc:407] Connection header received
0929 20:22:09.123905 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:09.123915 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:09.123990 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:09.124108 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:09.124859 (+ 751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37288: BlockingRecv error: recv got EOF from 127.0.0.1:37288 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:22:09.489797 18649 catalog_manager.cc:1259] Loaded cluster ID: 393dc759429d48e09cf591cec7ca2b1b
I20260929 20:22:09.489953 18649 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cfb3f0b9926d4307b9a8cf493557de06: loading cluster ID for follower catalog manager: success
I20260929 20:22:09.492796 18649 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cfb3f0b9926d4307b9a8cf493557de06: acquiring CA information for follower catalog manager: success
I20260929 20:22:09.494688 18649 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cfb3f0b9926d4307b9a8cf493557de06: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 20:22:09.652357 18715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:09.645114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37304 (local address 127.27.185.125:45305)
0929 20:22:09.645764 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:09.645770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:09.646077 (+ 307us) server_negotiation.cc:407] Connection header received
0929 20:22:09.646274 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:09.646278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:09.646437 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:09.646581 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:22:09.647073 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:09.647984 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:09.649934 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:09.650373 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:09.650489 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:09.651030 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:22:09.651687 (+ 657us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:22:09.651802 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:09.651942 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 20:22:09.652015 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":312,"threads_started":1}
I20260929 20:22:09.671228 18721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:09.664392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56194 (local address 127.27.185.124:33429)
0929 20:22:09.664949 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:09.664955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:09.665303 (+ 348us) server_negotiation.cc:407] Connection header received
0929 20:22:09.665580 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:09.665585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:09.665744 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:09.665896 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:22:09.666310 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:09.667042 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:09.668942 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:09.669372 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:09.669496 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:09.670013 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:22:09.670562 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:22:09.670678 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:09.670817 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 20:22:09.670885 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":281,"threads_started":1}
I20260929 20:22:09.712819 18728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:09.678293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39794 (local address 127.27.185.126:34175)
0929 20:22:09.703935 (+ 25642us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:09.703946 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:09.704817 (+ 871us) server_negotiation.cc:407] Connection header received
0929 20:22:09.705130 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:09.705138 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:09.705366 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:09.705564 (+ 198us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:22:09.706285 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:09.707496 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:09.709726 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:09.710434 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:09.710580 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:09.711245 (+ 665us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:22:09.711747 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:22:09.711871 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:09.712367 (+ 496us) server_negotiation.cc:299] Negotiation successful
0929 20:22:09.712475 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25437,"thread_start_us":25693,"threads_started":1}
I20260929 20:22:09.713048 18721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:09.678566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56196 (local address 127.27.185.124:33429)
0929 20:22:09.684551 (+ 5985us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:09.684560 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:09.705575 (+ 21015us) server_negotiation.cc:407] Connection header received
0929 20:22:09.705855 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:09.705863 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:09.705951 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:09.706038 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:22:09.707007 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:09.708067 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:09.710806 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:09.711478 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:09.711604 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:09.712246 (+ 642us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:22:09.712365 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:22:09.712427 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:09.712676 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 20:22:09.712759 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5793}
I20260929 20:22:09.717815 18715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:09.678055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37316 (local address 127.27.185.125:45305)
0929 20:22:09.678390 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:09.678403 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:09.710672 (+ 32269us) server_negotiation.cc:407] Connection header received
0929 20:22:09.711066 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:09.711074 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:09.711168 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:09.711242 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:22:09.712106 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:09.713288 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:09.715600 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:09.716334 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:09.716500 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:09.717028 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:22:09.717154 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:22:09.717267 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:09.717386 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 20:22:09.717481 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260929 20:22:09.718508 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 18361
I20260929 20:22:09.719594 18423 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:22:09.832805 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 18361
I20260929 20:22:09.854039 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 18436
W20260929 20:22:09.854024 18454 connection.cc:535] server connection from 127.0.0.1:37166 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 20:22:09.855170 18498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:22:09.961782 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 18436
I20260929 20:22:09.982863 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 18510
I20260929 20:22:09.984072 18572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:22:10.090595 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 18510
2026-09-29T20:22:10Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4967 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:22:10 dist-test-slave-4cx3 krb5kdc[18735](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:22:10 dist-test-slave-4cx3 krb5kdc[18735](info): set up 2 sockets
Sep 29 20:22:10 dist-test-slave-4cx3 krb5kdc[18735](info): commencing operation
krb5kdc: starting...
W20260929 20:22:11.206843 28389 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/krb5kdc/test-user.keytab.
Sep 29 20:22:11 dist-test-slave-4cx3 krb5kdc[18735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:22:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:22:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
Entry for principal kudu/127.27.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab.
I20260929 20:22:11.478633 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:33915
--webserver_interface=127.27.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:33915,127.27.185.125:35255,127.27.185.124:35739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:11.814121 18748 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 20:22:11.814544 18748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:11.814672 18748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:11.814810 18748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:11.822157 18748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:22:11.822302 18748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:11.822402 18748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:11.822503 18748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:22:11.822618 18748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:22:11.833362 18748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:33915,127.27.185.125:35255,127.27.185.124:35739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.126.keytab
--principal=kudu/127.27.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.126:33915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:11.834779 18748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:11.836162 18748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:11.848733 18753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:11.848738 18754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:11.849133 18756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:11.850193 18748 server_base.cc:1034] running on GCE node
I20260929 20:22:11.850999 18748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:11.852460 18748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:11.853688 18748 hybrid_clock.cc:648] HybridClock initialized: now 1790713331853619 us; error 37 us; skew 500 ppm
I20260929 20:22:11.854036 18748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:11 dist-test-slave-4cx3 krb5kdc[18735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713331, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:11.857198 18748 init.cc:377] Logged in from keytab as kudu/127.27.185.126@KRBTEST.COM (short username kudu)
I20260929 20:22:11.858657 18748 webserver.cc:460] Webserver started at http://127.27.185.126:40565/ using document root <none> and password file <none>
I20260929 20:22:11.859330 18748 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:11.859491 18748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:11.859783 18748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:11.862715 18748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-0/data/instance:
uuid: "f09baeed299641c5be81dcc0783d9fb2"
format_stamp: "Formatted at 2026-09-29 20:22:11 on dist-test-slave-4cx3"
server_key: "4bb2b2d42030a97c6f6d586976f94e9b"
server_key_iv: "6bf0d594beb5b33370a3229607e3250b"
server_key_version: "encryptionkey@0"
I20260929 20:22:11.863531 18748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-0/wal/instance:
uuid: "f09baeed299641c5be81dcc0783d9fb2"
format_stamp: "Formatted at 2026-09-29 20:22:11 on dist-test-slave-4cx3"
server_key: "4bb2b2d42030a97c6f6d586976f94e9b"
server_key_iv: "6bf0d594beb5b33370a3229607e3250b"
server_key_version: "encryptionkey@0"
I20260929 20:22:11.868836 18748 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:22:11.872484 18763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:11.873863 18748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:22:11.874010 18748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-0/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-0/wal
uuid: "f09baeed299641c5be81dcc0783d9fb2"
format_stamp: "Formatted at 2026-09-29 20:22:11 on dist-test-slave-4cx3"
server_key: "4bb2b2d42030a97c6f6d586976f94e9b"
server_key_iv: "6bf0d594beb5b33370a3229607e3250b"
server_key_version: "encryptionkey@0"
I20260929 20:22:11.874261 18748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:11.892158 18748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:11.896674 18748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:11.896909 18748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:11.916071 18748 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.126:33915
I20260929 20:22:11.916096 18816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.126:33915 every 8 connection(s)
I20260929 20:22:11.917481 18748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-0/data/info.pb
I20260929 20:22:11.921160 18817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:11.925343 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 18748
I20260929 20:22:11.925570 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-0/wal/instance
I20260929 20:22:11.926059 28389 external_mini_cluster.cc:1428] Setting key 619898fe0a1a8356454772435cd364b1
I20260929 20:22:11.928280 18817 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33915 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:22:11 dist-test-slave-4cx3 krb5kdc[18735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713331, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:22:11.952267 18820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:11.931455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48604 (local address 127.27.185.126:33915)
0929 20:22:11.933105 (+ 1650us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:11.933115 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:11.933175 (+ 60us) server_negotiation.cc:407] Connection header received
0929 20:22:11.934841 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:11.934847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:11.935086 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:11.935232 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:11.936490 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:11.937646 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:11.939873 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:11.940422 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:11.940560 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:11.944983 (+ 4423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:11.944991 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:11.945003 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:11.945032 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:11.947905 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:11.948810 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:11.948816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:11.948818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:11.948959 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:11.949473 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:11.949476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:11.949478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:11.949959 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:11.950136 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:11.950825 (+ 689us) server_negotiation.cc:299] Negotiation successful
0929 20:22:11.951133 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":1409,"spinlock_wait_cycles":19712,"thread_start_us":1213,"threads_started":1}
I20260929 20:22:11.952320 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:11.931452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33915 (local address 127.0.0.1:48604)
0929 20:22:11.932423 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:11.932494 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:11.934439 (+ 1945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:11.935340 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:11.935347 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:11.935982 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:11.936298 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:11.936314 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:11.937807 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:11.937816 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:11.939728 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:11.939745 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:11.940669 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:11.940678 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:11.940903 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:11.941630 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:11.941673 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:11.944716 (+ 3043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:11.948040 (+ 3324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:11.948054 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:11.948070 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:11.948669 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:11.949092 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:11.949100 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:11.949106 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:11.949352 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:11.950126 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:11.950135 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:11.950412 (+ 277us) client_negotiation.cc:769] Sending connection context
0929 20:22:11.950695 (+ 283us) client_negotiation.cc:240] Negotiation successful
0929 20:22:11.951320 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":81,"thread_start_us":379,"threads_started":1}
WARNING: no policy specified for kudu/127.27.185.125@KRBTEST.COM; defaulting to no policy
I20260929 20:22:11.955335 18817 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.125" port: 35255 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.125@KRBTEST.COM" created.
I20260929 20:22:11.956462 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:11.956083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:35255 (local address 127.0.0.1:56712)
0929 20:22:11.956259 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:11.956358 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.125:35255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 20:22:11.956743 18765 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.185.125:35255: connect: Connection refused (error 111)
I20260929 20:22:11.958213 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:11.957893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:35255 (local address 127.0.0.1:56714)
0929 20:22:11.958034 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:11.958097 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.125:35255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 20:22:11.958514 18817 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.125:35255: Network error: Client connection negotiation failed: client connection to 127.27.185.125:35255: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.125.keytab.
Entry for principal kudu/127.27.185.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.125.keytab.
I20260929 20:22:11.988288 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.125:35255
--webserver_interface=127.27.185.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.125.keytab
--principal=kudu/127.27.185.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:33915,127.27.185.125:35255,127.27.185.124:35739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:22:12.007882 18817 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.125" port: 35255 } attempt: 1
I20260929 20:22:12.009287 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.008661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:35255 (local address 127.0.0.1:56726)
0929 20:22:12.008919 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:12.009089 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.125:35255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 20:22:12.010437 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.009973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:35255 (local address 127.0.0.1:56738)
0929 20:22:12.010158 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:12.010259 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.125:35255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 20:22:12.010663 18817 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.125:35255: Network error: Client connection negotiation failed: client connection to 127.27.185.125:35255: connect: Connection refused (error 111)
I20260929 20:22:12.078940 18817 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.125" port: 35255 } attempt: 2
I20260929 20:22:12.080222 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.079678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:35255 (local address 127.0.0.1:56742)
0929 20:22:12.079965 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:12.080061 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.125:35255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 20:22:12.081414 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.081016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:35255 (local address 127.0.0.1:56758)
0929 20:22:12.081192 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:12.081279 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.125:35255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 20:22:12.081668 18817 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.125:35255: Network error: Client connection negotiation failed: client connection to 127.27.185.125:35255: connect: Connection refused (error 111)
I20260929 20:22:12.172951 18817 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.125" port: 35255 } attempt: 3
I20260929 20:22:12.174263 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.173758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:35255 (local address 127.0.0.1:56768)
0929 20:22:12.174028 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:12.174124 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.125:35255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260929 20:22:12.175540 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.175112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:35255 (local address 127.0.0.1:56782)
0929 20:22:12.175251 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:12.175322 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.125:35255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 20:22:12.175809 18817 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.125:35255: Network error: Client connection negotiation failed: client connection to 127.27.185.125:35255: connect: Connection refused (error 111)
I20260929 20:22:12.319103 18817 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.125" port: 35255 } attempt: 4
I20260929 20:22:12.320518 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.319946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:35255 (local address 127.0.0.1:35968)
0929 20:22:12.320275 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:12.320359 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.125:35255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
I20260929 20:22:12.321612 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.321300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:35255 (local address 127.0.0.1:35974)
0929 20:22:12.321426 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:12.321504 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.125:35255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 20:22:12.321997 18817 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.125:35255: Network error: Client connection negotiation failed: client connection to 127.27.185.125:35255: connect: Connection refused (error 111)
W20260929 20:22:12.328186 18824 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 20:22:12.328554 18824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:12.328682 18824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:12.328817 18824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:12.336328 18824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:22:12.336453 18824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:12.336578 18824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:12.336665 18824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:22:12.336774 18824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:22:12.346694 18824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:33915,127.27.185.125:35255,127.27.185.124:35739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.125.keytab
--principal=kudu/127.27.185.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.125:35255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.185.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:12.348111 18824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:12.349462 18824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:12.361306 18829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:12.361465 18832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:12.361526 18830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:12.362689 18824 server_base.cc:1034] running on GCE node
I20260929 20:22:12.363615 18824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:12.365120 18824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:12.366338 18824 hybrid_clock.cc:648] HybridClock initialized: now 1790713332366242 us; error 68 us; skew 500 ppm
I20260929 20:22:12.366679 18824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:12 dist-test-slave-4cx3 krb5kdc[18735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713332, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:12.370107 18824 init.cc:377] Logged in from keytab as kudu/127.27.185.125@KRBTEST.COM (short username kudu)
I20260929 20:22:12.371798 18824 webserver.cc:460] Webserver started at http://127.27.185.125:46243/ using document root <none> and password file <none>
I20260929 20:22:12.372521 18824 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:12.372658 18824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:12.372994 18824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:12.375993 18824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-1/data/instance:
uuid: "d770e76d9ec84a25b51773530e4135ef"
format_stamp: "Formatted at 2026-09-29 20:22:12 on dist-test-slave-4cx3"
server_key: "813dbcc739ff2e2fa96f94584cff175a"
server_key_iv: "1d8506a25d77a3261db2fbbbe572ed51"
server_key_version: "encryptionkey@0"
I20260929 20:22:12.376786 18824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-1/wal/instance:
uuid: "d770e76d9ec84a25b51773530e4135ef"
format_stamp: "Formatted at 2026-09-29 20:22:12 on dist-test-slave-4cx3"
server_key: "813dbcc739ff2e2fa96f94584cff175a"
server_key_iv: "1d8506a25d77a3261db2fbbbe572ed51"
server_key_version: "encryptionkey@0"
I20260929 20:22:12.382007 18824 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:22:12.385564 18840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:12.386891 18824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 20:22:12.387089 18824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-1/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-1/wal
uuid: "d770e76d9ec84a25b51773530e4135ef"
format_stamp: "Formatted at 2026-09-29 20:22:12 on dist-test-slave-4cx3"
server_key: "813dbcc739ff2e2fa96f94584cff175a"
server_key_iv: "1d8506a25d77a3261db2fbbbe572ed51"
server_key_version: "encryptionkey@0"
I20260929 20:22:12.387308 18824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:12.406162 18824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:12.410727 18824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:12.411029 18824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:12.429677 18824 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.125:35255
I20260929 20:22:12.429741 18891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.125:35255 every 8 connection(s)
I20260929 20:22:12.431052 18824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-1/data/info.pb
I20260929 20:22:12.433991 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 18824
I20260929 20:22:12.434185 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-1/wal/instance
I20260929 20:22:12.434715 28389 external_mini_cluster.cc:1428] Setting key ab1796ed13d504058345be7266d53d70
I20260929 20:22:12.435175 18892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:12.445271 18892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33915 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:22:12 dist-test-slave-4cx3 krb5kdc[18735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713332, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.125@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.185.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.185.124@KRBTEST.COM" created.
I20260929 20:22:12.466723 18820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.448226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49558 (local address 127.27.185.126:33915)
0929 20:22:12.448400 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:12.448405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:12.449728 (+ 1323us) server_negotiation.cc:407] Connection header received
0929 20:22:12.451697 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:12.451703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:12.451796 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:12.451908 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:12.453457 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:12.454149 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:12.455790 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:12.456224 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:12.456368 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:12.460054 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:12.460059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:12.460062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:12.460097 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:12.463114 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:12.464008 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:12.464012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:12.464014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:12.464175 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:12.464675 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:12.464681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:12.464684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:12.465134 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:12.465297 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:12.466032 (+ 735us) server_negotiation.cc:299] Negotiation successful
0929 20:22:12.466300 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:22:12.467474 18895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.448771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33915 (local address 127.0.0.1:49558)
0929 20:22:12.449549 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:12.449596 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:12.451360 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:12.452155 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:12.452164 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:12.452689 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:12.453266 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:12.453283 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:12.454322 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:12.454327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:12.455638 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:12.455651 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:12.456469 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:12.456477 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:12.456719 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:12.457529 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:12.457566 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:12.459812 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:12.463312 (+ 3500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:12.463319 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:12.463334 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:12.463858 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:12.464303 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:12.464311 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:12.464317 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:12.464540 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:12.465268 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:12.465274 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:12.465622 (+ 348us) client_negotiation.cc:769] Sending connection context
0929 20:22:12.465851 (+ 229us) client_negotiation.cc:240] Negotiation successful
0929 20:22:12.466249 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":325,"threads_started":1}
I20260929 20:22:12.469264 18892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.125" port: 35255 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:22:12 dist-test-slave-4cx3 krb5kdc[18735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713332, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.125@KRBTEST.COM for kudu/127.27.185.125@KRBTEST.COM
I20260929 20:22:12.485546 18895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.470122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:35255 (local address 127.0.0.1:35988)
0929 20:22:12.470291 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:12.470310 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:12.470414 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:12.471479 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:12.471483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:12.471934 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:12.472060 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:12.472068 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:12.473431 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:12.473439 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:12.474281 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:12.474291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:12.475109 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:12.475114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:12.475306 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:12.475943 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:12.475962 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:12.478198 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:12.482959 (+ 4761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:12.482964 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:12.482966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:12.483340 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:12.483843 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:12.483852 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:12.483855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:12.483994 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:12.484799 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:12.484802 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:12.484928 (+ 126us) client_negotiation.cc:769] Sending connection context
0929 20:22:12.485034 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 20:22:12.485245 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 20:22:12.486085 18896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.470372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35988 (local address 127.27.185.125:35255)
0929 20:22:12.471040 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:12.471046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:12.471070 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:22:12.471112 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:12.471117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:12.471291 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:12.471485 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:12.472228 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:12.473295 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:12.474418 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:12.474968 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:12.475105 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:12.478453 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:12.478471 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:12.478493 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:12.478536 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:12.482780 (+ 4244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:12.483549 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:12.483562 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:12.483568 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:12.483720 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:12.484123 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:12.484131 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:12.484137 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:12.484663 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:12.484879 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:12.485247 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 20:22:12.485574 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":383,"threads_started":1}
I20260929 20:22:12.488715 18892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.124" port: 35739 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:22:12.489943 18895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.489371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:35739 (local address 127.0.0.1:50234)
0929 20:22:12.489595 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:12.489747 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260929 20:22:12.490245 18841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.185.124:35739: connect: Connection refused (error 111)
I20260929 20:22:12.492089 18895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.491597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:35739 (local address 127.0.0.1:50248)
0929 20:22:12.491806 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:12.491948 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 20:22:12.492324 18892 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.124:35739: Network error: Client connection negotiation failed: client connection to 127.27.185.124:35739: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.185.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.124.keytab.
Entry for principal kudu/127.27.185.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.124.keytab.
I20260929 20:22:12.503918 28389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
/tmp/dist-test-taskPFKKvj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.124:35739
--webserver_interface=127.27.185.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.124.keytab
--principal=kudu/127.27.185.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.185.84:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.185.126:33915,127.27.185.125:35255,127.27.185.124:35739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:22:12.541682 18892 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.124" port: 35739 } attempt: 1
I20260929 20:22:12.542814 18895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.542356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:35739 (local address 127.0.0.1:50250)
0929 20:22:12.542549 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:12.542667 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 20:22:12.544018 18895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.543626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:35739 (local address 127.0.0.1:50262)
0929 20:22:12.543809 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:12.543910 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 20:22:12.544230 18892 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.124:35739: Network error: Client connection negotiation failed: client connection to 127.27.185.124:35739: connect: Connection refused (error 111)
I20260929 20:22:12.612481 18892 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.124" port: 35739 } attempt: 2
I20260929 20:22:12.613701 18895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.613256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:35739 (local address 127.0.0.1:50274)
0929 20:22:12.613452 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:12.613546 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 20:22:12.614816 18895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.614431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:35739 (local address 127.0.0.1:50290)
0929 20:22:12.614634 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:12.614702 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260929 20:22:12.615046 18892 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.124:35739: Network error: Client connection negotiation failed: client connection to 127.27.185.124:35739: connect: Connection refused (error 111)
I20260929 20:22:12.621212 18817 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.125" port: 35255 } attempt: 5
Sep 29 20:22:12 dist-test-slave-4cx3 krb5kdc[18735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713331, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for kudu/127.27.185.125@KRBTEST.COM
I20260929 20:22:12.634773 18896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.621980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35990 (local address 127.27.185.125:35255)
0929 20:22:12.622162 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:12.622170 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:12.622271 (+ 101us) server_negotiation.cc:407] Connection header received
0929 20:22:12.622498 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:12.622502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:12.622580 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:12.622688 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:12.623497 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:12.624197 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:12.625400 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:12.625857 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:12.625997 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:12.629226 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:12.629231 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:12.629235 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:12.629267 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:12.632134 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:12.632823 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:12.632830 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:12.632834 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:12.632957 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:12.633338 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:12.633345 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:12.633348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:12.633649 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:12.633805 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:12.634177 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 20:22:12.634414 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 20:22:12.634770 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.621957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:35255 (local address 127.0.0.1:35990)
0929 20:22:12.622162 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:12.622183 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:12.622348 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:12.622721 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:12.622726 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:12.623136 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:12.623319 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:12.623328 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:12.624336 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:12.624342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:12.625243 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:12.625255 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:12.626018 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:12.626024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:12.626228 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:12.626857 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:12.626875 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:12.629022 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:12.632305 (+ 3283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:12.632310 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:12.632314 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:12.632698 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:12.633084 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:12.633089 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:12.633093 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:12.633229 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:12.633767 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:12.633772 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:12.633926 (+ 154us) client_negotiation.cc:769] Sending connection context
0929 20:22:12.634083 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 20:22:12.634415 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 20:22:12.636132 18817 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.124" port: 35739 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:22:12.637238 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.636828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:35739 (local address 127.0.0.1:50300)
0929 20:22:12.636995 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:12.637071 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 20:22:12.638157 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.637833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:35739 (local address 127.0.0.1:50302)
0929 20:22:12.637976 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:12.638058 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 20:22:12.638381 18817 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.124:35739: Network error: Client connection negotiation failed: client connection to 127.27.185.124:35739: connect: Connection refused (error 111)
I20260929 20:22:12.689586 18817 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.124" port: 35739 } attempt: 1
I20260929 20:22:12.690786 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.690329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:35739 (local address 127.0.0.1:50314)
0929 20:22:12.690525 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:12.690622 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 20:22:12.691993 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.691583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:35739 (local address 127.0.0.1:50316)
0929 20:22:12.691730 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:12.691842 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 20:22:12.692216 18817 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.124:35739: Network error: Client connection negotiation failed: client connection to 127.27.185.124:35739: connect: Connection refused (error 111)
I20260929 20:22:12.706255 18892 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.124" port: 35739 } attempt: 3
I20260929 20:22:12.707450 18895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.706913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:35739 (local address 127.0.0.1:50330)
0929 20:22:12.707136 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:12.707244 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 20:22:12.708665 18895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.708209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:35739 (local address 127.0.0.1:50346)
0929 20:22:12.708431 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:12.708549 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260929 20:22:12.708861 18892 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.124:35739: Network error: Client connection negotiation failed: client connection to 127.27.185.124:35739: connect: Connection refused (error 111)
I20260929 20:22:12.760453 18817 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.124" port: 35739 } attempt: 2
I20260929 20:22:12.761693 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.761186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:35739 (local address 127.0.0.1:50362)
0929 20:22:12.761438 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:12.761527 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 20:22:12.762867 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.762492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:35739 (local address 127.0.0.1:50378)
0929 20:22:12.762645 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:12.762729 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 20:22:12.763084 18817 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.124:35739: Network error: Client connection negotiation failed: client connection to 127.27.185.124:35739: connect: Connection refused (error 111)
W20260929 20:22:12.848829 18899 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 20:22:12.849225 18899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:12.849378 18899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:12.849517 18899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 20:22:12.852126 18892 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.124" port: 35739 } attempt: 4
I20260929 20:22:12.853322 18895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.852781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:35739 (local address 127.0.0.1:50386)
0929 20:22:12.853049 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:12.853151 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 20:22:12.854343 18895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.854012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:35739 (local address 127.0.0.1:50394)
0929 20:22:12.854177 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:12.854241 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 20:22:12.854583 18892 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.124:35739: Network error: Client connection negotiation failed: client connection to 127.27.185.124:35739: connect: Connection refused (error 111)
W20260929 20:22:12.857901 18899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:22:12.858036 18899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:12.858157 18899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:12.858245 18899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:22:12.858352 18899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:22:12.868745 18899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.185.84:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.185.126:33915,127.27.185.125:35255,127.27.185.124:35739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/krb5kdc/kudu_127.27.185.124.keytab
--principal=kudu/127.27.185.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.185.124:35739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.185.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:12.869282 18817 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.124" port: 35739 } attempt: 3
I20260929 20:22:12.870499 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.869936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:35739 (local address 127.0.0.1:50396)
0929 20:22:12.870199 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:12.870329 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 20:22:12.870522 18899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:12.871788 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.871263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:35739 (local address 127.0.0.1:50404)
0929 20:22:12.871495 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:12.871622 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.185.124:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260929 20:22:12.872048 18817 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.185.124:35739: Network error: Client connection negotiation failed: client connection to 127.27.185.124:35739: connect: Connection refused (error 111)
I20260929 20:22:12.872659 18899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:12.885006 18905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:12.884940 18904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:12.885598 18907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:12.886847 18899 server_base.cc:1034] running on GCE node
I20260929 20:22:12.887821 18899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:12.889286 18899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:12.890571 18899 hybrid_clock.cc:648] HybridClock initialized: now 1790713332890501 us; error 42 us; skew 500 ppm
I20260929 20:22:12.890964 18899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:12 dist-test-slave-4cx3 krb5kdc[18735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713332, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:12.894456 18899 init.cc:377] Logged in from keytab as kudu/127.27.185.124@KRBTEST.COM (short username kudu)
I20260929 20:22:12.896159 18899 webserver.cc:460] Webserver started at http://127.27.185.124:38095/ using document root <none> and password file <none>
I20260929 20:22:12.896921 18899 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:12.897055 18899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:12.897384 18899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:12.900491 18899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-2/data/instance:
uuid: "9780811cd26f4c31aec7db0006201fbe"
format_stamp: "Formatted at 2026-09-29 20:22:12 on dist-test-slave-4cx3"
server_key: "4bad548ce40a59bb514349114272a2f8"
server_key_iv: "46a842f9871bd1d048ddde470dbf9c4b"
server_key_version: "encryptionkey@0"
I20260929 20:22:12.901284 18899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-2/wal/instance:
uuid: "9780811cd26f4c31aec7db0006201fbe"
format_stamp: "Formatted at 2026-09-29 20:22:12 on dist-test-slave-4cx3"
server_key: "4bad548ce40a59bb514349114272a2f8"
server_key_iv: "46a842f9871bd1d048ddde470dbf9c4b"
server_key_version: "encryptionkey@0"
I20260929 20:22:12.907091 18899 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:22:12.910703 18915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:12.912154 18899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:22:12.912343 18899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-2/data,/tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-2/wal
uuid: "9780811cd26f4c31aec7db0006201fbe"
format_stamp: "Formatted at 2026-09-29 20:22:12 on dist-test-slave-4cx3"
server_key: "4bad548ce40a59bb514349114272a2f8"
server_key_iv: "46a842f9871bd1d048ddde470dbf9c4b"
server_key_version: "encryptionkey@0"
I20260929 20:22:12.912585 18899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:12.932384 18899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:12.936728 18899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:12.937033 18899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:12.956089 18899 rpc_server.cc:307] RPC server started. Bound to: 127.27.185.124:35739
I20260929 20:22:12.956182 18966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.185.124:35739 every 8 connection(s)
I20260929 20:22:12.957484 18899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-2/data/info.pb
I20260929 20:22:12.961346 18967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:12.963590 28389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu as pid 18899
I20260929 20:22:12.963778 28389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPFKKvj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790712966917858-28389-0/minicluster-data/master-2/wal/instance
I20260929 20:22:12.964342 28389 external_mini_cluster.cc:1428] Setting key 61877ea6ce2073917b69633b685888d2
I20260929 20:22:12.970921 18967 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33915 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:22:12 dist-test-slave-4cx3 krb5kdc[18735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.124@KRBTEST.COM
Sep 29 20:22:12 dist-test-slave-4cx3 krb5kdc[18735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713332, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.124@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:22:12.988041 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.966212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50420 (local address 127.27.185.124:35739)
0929 20:22:12.967051 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:12.967066 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:12.967108 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:22:12.968722 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:12.968746 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:12.969229 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:12.969677 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:12.971090 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:12.972484 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:12.974345 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:12.975353 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:12.975585 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:12.979950 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:12.979960 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:12.979974 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:12.980008 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:12.983225 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:12.984003 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:12.984009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:12.984016 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:12.984223 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:12.984624 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:12.984632 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:12.984636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:12.985426 (+ 790us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:12.985758 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:12.986223 (+ 465us) server_negotiation.cc:299] Negotiation successful
0929 20:22:12.986696 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":366,"threads_started":1}
I20260929 20:22:12.989758 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.973507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49566 (local address 127.27.185.126:33915)
0929 20:22:12.974108 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:12.974115 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:12.974611 (+ 496us) server_negotiation.cc:407] Connection header received
0929 20:22:12.974793 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:12.974797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:12.974869 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:12.974963 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:12.976308 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:12.977116 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:12.978577 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:12.979215 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:12.979316 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:12.983359 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:12.983366 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:12.983371 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:12.983434 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:12.986663 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:12.987491 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:12.987497 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:12.987500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:12.987658 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:12.988108 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:12.988115 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:12.988120 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:12.988465 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:12.988637 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:12.989103 (+ 466us) server_negotiation.cc:299] Negotiation successful
0929 20:22:12.989340 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":296,"threads_started":1}
I20260929 20:22:12.989765 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.973527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.126:33915 (local address 127.0.0.1:49566)
0929 20:22:12.974450 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:12.974490 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:12.974669 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:12.975046 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:12.975052 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:12.975919 (+ 867us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:12.976145 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:12.976154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:12.977271 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:12.977276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:12.978424 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:12.978439 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:12.979494 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:12.979504 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:12.979702 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:12.980421 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:12.980438 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:12.983139 (+ 2701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:12.986845 (+ 3706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:12.986854 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:12.986860 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:12.987305 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:12.987816 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:12.987820 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:12.987822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:12.987964 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:12.988615 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:12.988623 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:12.988803 (+ 180us) client_negotiation.cc:769] Sending connection context
0929 20:22:12.988974 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 20:22:12.989296 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":532,"threads_started":1}
I20260929 20:22:12.991375 18967 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.125" port: 35255 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:22:12 dist-test-slave-4cx3 krb5kdc[18735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713332, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.124@KRBTEST.COM for kudu/127.27.185.125@KRBTEST.COM
Sep 29 20:22:12 dist-test-slave-4cx3 krb5kdc[18735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.126@KRBTEST.COM
I20260929 20:22:13.007867 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.992219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.125:35255 (local address 127.0.0.1:36006)
0929 20:22:12.992484 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:12.992502 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:12.992609 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:12.992919 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:12.992924 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:12.993396 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:12.993607 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:12.993620 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:12.994998 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:12.995003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:12.996439 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:12.996464 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:12.997481 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:12.997487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:12.997689 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:12.998380 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:12.998404 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:13.001232 (+ 2828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:13.005251 (+ 4019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:13.005255 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:13.005258 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:13.005634 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:13.006091 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:13.006095 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:13.006098 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:13.006239 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:13.006801 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:13.006806 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:13.006972 (+ 166us) client_negotiation.cc:769] Sending connection context
0929 20:22:13.007124 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 20:22:13.007477 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260929 20:22:13.007898 18896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.992190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36006 (local address 127.27.185.125:35255)
0929 20:22:12.992433 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:12.992437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:12.992570 (+ 133us) server_negotiation.cc:407] Connection header received
0929 20:22:12.992716 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:12.992720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:12.992785 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:12.992863 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:12.993780 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:12.994854 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:12.996618 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:12.997274 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:12.997471 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:13.001456 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:13.001461 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:13.001465 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:13.001506 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:13.005082 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:13.005763 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:13.005768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:13.005772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:13.005938 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:13.006369 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:13.006374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:13.006378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:13.006657 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:13.006828 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:13.007225 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 20:22:13.007488 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 20:22:13.008420 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:12.991389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49570 (local address 127.27.185.126:33915)
0929 20:22:12.991564 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:12.991571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:12.991696 (+ 125us) server_negotiation.cc:407] Connection header received
0929 20:22:12.991862 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:12.991866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:12.991983 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:12.992105 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:12.993802 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:12.994855 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:12.996618 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:12.997290 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:12.997421 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:13.002309 (+ 4888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:13.002318 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:13.002324 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:13.002418 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:13.005763 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:13.006426 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:13.006432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:13.006436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:13.006568 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:13.006937 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:13.006943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:13.006946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:13.007218 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:13.007543 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:13.007872 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 20:22:13.008088 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 20:22:13.009217 18967 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.185.124" port: 35739 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:22:13 dist-test-slave-4cx3 krb5kdc[18735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713332, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.124@KRBTEST.COM for kudu/127.27.185.124@KRBTEST.COM
Sep 29 20:22:13 dist-test-slave-4cx3 krb5kdc[18735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.185.125@KRBTEST.COM
I20260929 20:22:13.023576 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:13.010176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50430 (local address 127.27.185.124:35739)
0929 20:22:13.010336 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:13.010341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:13.010359 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:22:13.010550 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:13.010555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:13.010631 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:13.010724 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:13.011405 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:13.012106 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:13.013218 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:13.013658 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:13.013793 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:13.017377 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:13.017384 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:13.017389 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:13.017427 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:13.020770 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:13.021554 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:13.021562 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:13.021566 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:13.021721 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:13.022072 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:13.022080 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:13.022083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:13.022463 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:13.022620 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:13.022953 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 20:22:13.023236 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":54}
I20260929 20:22:13.023600 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:13.010012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:35739 (local address 127.0.0.1:50430)
0929 20:22:13.010196 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:13.010215 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:13.010393 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:13.010738 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:13.010742 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:13.011129 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:13.011250 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:13.011258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:13.012229 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:13.012238 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:13.013081 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:13.013091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:13.013799 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:13.013804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:13.014026 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:13.014604 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:13.014620 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:13.017168 (+ 2548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:13.020922 (+ 3754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:13.020926 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:13.020929 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:13.021425 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:13.021836 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:13.021839 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:13.021841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:13.021963 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:13.022582 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:13.022585 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:13.022706 (+ 121us) client_negotiation.cc:769] Sending connection context
0929 20:22:13.022819 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 20:22:13.023090 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":4}
I20260929 20:22:13.026104 18896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:13.010567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36016 (local address 127.27.185.125:35255)
0929 20:22:13.010759 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:13.010762 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:13.010775 (+ 13us) server_negotiation.cc:407] Connection header received
0929 20:22:13.010811 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:13.010815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:13.010868 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:13.010970 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:13.012198 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:13.012895 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:13.014076 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:13.014487 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:13.014620 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:13.019446 (+ 4826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:13.019454 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:13.019459 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:13.019504 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:13.022955 (+ 3451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:13.024016 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:13.024024 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:13.024028 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:13.024193 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:13.024599 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:13.024603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:13.024606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:13.024996 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:13.025212 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:13.025649 (+ 437us) server_negotiation.cc:299] Negotiation successful
0929 20:22:13.025827 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:22:13.029575 18967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9780811cd26f4c31aec7db0006201fbe: Bootstrap starting.
I20260929 20:22:13.031141 28389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:22:13.033507 18967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9780811cd26f4c31aec7db0006201fbe: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:13.034646 18967 log.cc:824] T 00000000000000000000000000000000 P 9780811cd26f4c31aec7db0006201fbe: Log is configured to *not* fsync() on all Append() calls
I20260929 20:22:13.034245 18790 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f09baeed299641c5be81dcc0783d9fb2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49570
I20260929 20:22:13.037568 18967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9780811cd26f4c31aec7db0006201fbe: No bootstrap required, opened a new log
I20260929 20:22:13.044711 18967 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9780811cd26f4c31aec7db0006201fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09baeed299641c5be81dcc0783d9fb2" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33915 } } peers { permanent_uuid: "d770e76d9ec84a25b51773530e4135ef" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 35255 } } peers { permanent_uuid: "9780811cd26f4c31aec7db0006201fbe" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 35739 } }
I20260929 20:22:13.045090 18967 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9780811cd26f4c31aec7db0006201fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9780811cd26f4c31aec7db0006201fbe, State: Initialized, Role: FOLLOWER
I20260929 20:22:13.045533 18967 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9780811cd26f4c31aec7db0006201fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09baeed299641c5be81dcc0783d9fb2" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33915 } } peers { permanent_uuid: "d770e76d9ec84a25b51773530e4135ef" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 35255 } } peers { permanent_uuid: "9780811cd26f4c31aec7db0006201fbe" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 35739 } }
I20260929 20:22:13.046489 18974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9780811cd26f4c31aec7db0006201fbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09baeed299641c5be81dcc0783d9fb2" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33915 } } peers { permanent_uuid: "d770e76d9ec84a25b51773530e4135ef" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 35255 } } peers { permanent_uuid: "9780811cd26f4c31aec7db0006201fbe" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 35739 } } }
I20260929 20:22:13.046882 18974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9780811cd26f4c31aec7db0006201fbe [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:22:13.047533 18967 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9780811cd26f4c31aec7db0006201fbe [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:22:13.049252 18817 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.124" port: 35739 } attempt: 4
W20260929 20:22:13.058470 18986 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9780811cd26f4c31aec7db0006201fbe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:22:13.058607 18986 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 29 20:22:13 dist-test-slave-4cx3 krb5kdc[18735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713331, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.126@KRBTEST.COM for kudu/127.27.185.124@KRBTEST.COM
I20260929 20:22:13.067893 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:13.050040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:35739 (local address 127.0.0.1:50436)
0929 20:22:13.050220 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:13.050239 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:13.050368 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:13.050712 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:13.050717 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:13.051116 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:13.051281 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:13.051290 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:13.052872 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:13.052879 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:13.054349 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:13.054367 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:13.055867 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:13.055873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:13.056079 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:13.056799 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:13.056822 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:13.059783 (+ 2961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:13.064593 (+ 4810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:13.064600 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:13.064604 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:13.065152 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:13.065631 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:13.065636 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:13.065639 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:13.065814 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:13.066727 (+ 913us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:13.066735 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:13.066969 (+ 234us) client_negotiation.cc:769] Sending connection context
0929 20:22:13.067135 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 20:22:13.067477 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 20:22:13.068450 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:13.050105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50436 (local address 127.27.185.124:35739)
0929 20:22:13.050399 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:13.050404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:13.050422 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:22:13.050474 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:13.050480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:13.050543 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:13.050672 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:13.051506 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:13.052721 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:13.054536 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:13.055255 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:13.055399 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:13.059977 (+ 4578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:13.059988 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:13.059994 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:13.060028 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:13.064339 (+ 4311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:13.065301 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:13.065306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:13.065310 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:13.065507 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:13.066052 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:13.066059 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:13.066062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:13.066585 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:13.066778 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:13.067361 (+ 583us) server_negotiation.cc:299] Negotiation successful
0929 20:22:13.067775 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260929 20:22:13.057406 18899 master_runner.cc:186] Error getting master registration for 127.27.185.126:33915: OK, instance_id { permanent_uuid: "f09baeed299641c5be81dcc0783d9fb2" instance_seqno: 1790713331910477 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 20:22:13.069689 18899 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 20:22:13.073508 18817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09baeed299641c5be81dcc0783d9fb2: Bootstrap starting.
I20260929 20:22:13.077623 18817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f09baeed299641c5be81dcc0783d9fb2: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:13.078981 18817 log.cc:824] T 00000000000000000000000000000000 P f09baeed299641c5be81dcc0783d9fb2: Log is configured to *not* fsync() on all Append() calls
I20260929 20:22:13.082460 18817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09baeed299641c5be81dcc0783d9fb2: No bootstrap required, opened a new log
I20260929 20:22:13.090615 18817 raft_consensus.cc:348] T 00000000000000000000000000000000 P f09baeed299641c5be81dcc0783d9fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09baeed299641c5be81dcc0783d9fb2" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33915 } } peers { permanent_uuid: "d770e76d9ec84a25b51773530e4135ef" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 35255 } } peers { permanent_uuid: "9780811cd26f4c31aec7db0006201fbe" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 35739 } }
I20260929 20:22:13.091032 18817 raft_consensus.cc:729] T 00000000000000000000000000000000 P f09baeed299641c5be81dcc0783d9fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f09baeed299641c5be81dcc0783d9fb2, State: Initialized, Role: FOLLOWER
I20260929 20:22:13.091625 18817 consensus_queue.cc:260] T 00000000000000000000000000000000 P f09baeed299641c5be81dcc0783d9fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09baeed299641c5be81dcc0783d9fb2" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33915 } } peers { permanent_uuid: "d770e76d9ec84a25b51773530e4135ef" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 35255 } } peers { permanent_uuid: "9780811cd26f4c31aec7db0006201fbe" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 35739 } }
I20260929 20:22:13.092883 18989 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09baeed299641c5be81dcc0783d9fb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09baeed299641c5be81dcc0783d9fb2" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33915 } } peers { permanent_uuid: "d770e76d9ec84a25b51773530e4135ef" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 35255 } } peers { permanent_uuid: "9780811cd26f4c31aec7db0006201fbe" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 35739 } } }
I20260929 20:22:13.093570 18989 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09baeed299641c5be81dcc0783d9fb2 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:22:13.094177 18817 sys_catalog.cc:564] T 00000000000000000000000000000000 P f09baeed299641c5be81dcc0783d9fb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:22:13.094265 18790 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09baeed299641c5be81dcc0783d9fb2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 20:22:13.094442 18790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f09baeed299641c5be81dcc0783d9fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:13.098675 18790 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09baeed299641c5be81dcc0783d9fb2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09baeed299641c5be81dcc0783d9fb2" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33915 } } peers { permanent_uuid: "d770e76d9ec84a25b51773530e4135ef" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 35255 } } peers { permanent_uuid: "9780811cd26f4c31aec7db0006201fbe" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 35739 } }
I20260929 20:22:13.102867 18942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09baeed299641c5be81dcc0783d9fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9780811cd26f4c31aec7db0006201fbe"
I20260929 20:22:13.103438 18942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9780811cd26f4c31aec7db0006201fbe [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:13.107147 18942 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9780811cd26f4c31aec7db0006201fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f09baeed299641c5be81dcc0783d9fb2 in term 1.
I20260929 20:22:13.108026 18764 leader_election.cc:302] T 00000000000000000000000000000000 P f09baeed299641c5be81dcc0783d9fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9780811cd26f4c31aec7db0006201fbe, f09baeed299641c5be81dcc0783d9fb2; no voters:
I20260929 20:22:13.108639 18790 leader_election.cc:288] T 00000000000000000000000000000000 P f09baeed299641c5be81dcc0783d9fb2 [CANDIDATE]: Term 1 election: Requested vote from peers d770e76d9ec84a25b51773530e4135ef (127.27.185.125:35255), 9780811cd26f4c31aec7db0006201fbe (127.27.185.124:35739)
I20260929 20:22:13.104331 18748 master_runner.cc:186] Error getting master registration for 127.27.185.125:35255: OK, instance_id { permanent_uuid: "d770e76d9ec84a25b51773530e4135ef" instance_seqno: 1790713332424371 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 20:22:13.116724 18748 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 20:22:13.116906 18989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f09baeed299641c5be81dcc0783d9fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:22:13.117900 18989 raft_consensus.cc:686] T 00000000000000000000000000000000 P f09baeed299641c5be81dcc0783d9fb2 [term 1 LEADER]: Becoming Leader. State: Replica: f09baeed299641c5be81dcc0783d9fb2, State: Running, Role: LEADER
I20260929 20:22:13.118491 18989 consensus_queue.cc:237] T 00000000000000000000000000000000 P f09baeed299641c5be81dcc0783d9fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09baeed299641c5be81dcc0783d9fb2" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33915 } } peers { permanent_uuid: "d770e76d9ec84a25b51773530e4135ef" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 35255 } } peers { permanent_uuid: "9780811cd26f4c31aec7db0006201fbe" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 35739 } }
I20260929 20:22:13.126000 19003 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09baeed299641c5be81dcc0783d9fb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f09baeed299641c5be81dcc0783d9fb2. Latest consensus state: current_term: 1 leader_uuid: "f09baeed299641c5be81dcc0783d9fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09baeed299641c5be81dcc0783d9fb2" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33915 } } peers { permanent_uuid: "d770e76d9ec84a25b51773530e4135ef" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 35255 } } peers { permanent_uuid: "9780811cd26f4c31aec7db0006201fbe" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 35739 } } }
I20260929 20:22:13.126982 19003 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09baeed299641c5be81dcc0783d9fb2 [sys.catalog]: This master's current role is: LEADER
W20260929 20:22:13.139180 19000 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f09baeed299641c5be81dcc0783d9fb2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:22:13.139586 19000 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:22:13.142656 19007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:22:13.146245 19007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:22:13.153769 18892 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.185.124" port: 35739 } attempt: 5
W20260929 20:22:13.149806 18769 tablet.cc:2369] T 00000000000000000000000000000000 P f09baeed299641c5be81dcc0783d9fb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 20:22:13 dist-test-slave-4cx3 krb5kdc[18735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713332, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.185.125@KRBTEST.COM for kudu/127.27.185.124@KRBTEST.COM
I20260929 20:22:13.177812 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:13.138506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50450 (local address 127.27.185.124:35739)
0929 20:22:13.138710 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:13.138719 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:13.149149 (+ 10430us) server_negotiation.cc:407] Connection header received
0929 20:22:13.149331 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:13.149337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:13.149433 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:13.149538 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:13.157912 (+ 8374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:13.159123 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:13.161110 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:13.161843 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:13.161937 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:13.165834 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:13.165842 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:13.165847 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:13.165891 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:13.171910 (+ 6019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:13.173296 (+ 1386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:13.173303 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:13.173306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:13.173578 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:13.174380 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:13.174389 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:13.174392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:13.175075 (+ 683us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:13.175826 (+ 751us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:13.176530 (+ 704us) server_negotiation.cc:299] Negotiation successful
0929 20:22:13.177028 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:22:13.183715 18895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:13.154329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.185.124:35739 (local address 127.0.0.1:50466)
0929 20:22:13.154696 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:13.154717 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:13.154815 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:13.156316 (+ 1501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:13.156320 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:13.156849 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:13.157046 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:13.157057 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:13.162885 (+ 5828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:13.162891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:13.164643 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:13.164659 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:13.169664 (+ 5005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:13.169672 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:13.169907 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:13.170486 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:13.170511 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:13.173457 (+ 2946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:13.180856 (+ 7399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:13.180860 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:13.180864 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:13.181316 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:13.181792 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:13.181797 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:13.181802 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:13.182012 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:13.182672 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:13.182677 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:13.182835 (+ 158us) client_negotiation.cc:769] Sending connection context
0929 20:22:13.182979 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 20:22:13.183277 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249}
I20260929 20:22:13.184473 18896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:13.133990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36022 (local address 127.27.185.125:35255)
0929 20:22:13.134207 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:13.134213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:13.138358 (+ 4145us) server_negotiation.cc:407] Connection header received
0929 20:22:13.142922 (+ 4564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:13.142932 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:13.143050 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:13.143140 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:13.152342 (+ 9202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:13.153415 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:13.161791 (+ 8376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:13.162476 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:13.162592 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:13.174174 (+ 11582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:13.174181 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:13.174185 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:13.174233 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:13.178071 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:13.180084 (+ 2013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:13.180092 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:13.180098 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:13.180263 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:13.180766 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:13.180773 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:13.180779 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:13.181206 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:13.181467 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:13.183880 (+ 2413us) server_negotiation.cc:299] Negotiation successful
0929 20:22:13.184144 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 20:22:13.184964 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:13.154810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50466 (local address 127.27.185.124:35739)
0929 20:22:13.155774 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:13.155780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:13.155831 (+ 51us) server_negotiation.cc:407] Connection header received
0929 20:22:13.155913 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:13.155920 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:13.156076 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:13.157234 (+ 1158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:13.157652 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:13.162516 (+ 4864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:13.165074 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:13.166928 (+ 1854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:13.170219 (+ 3291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:13.173737 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:13.173744 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:13.173749 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:13.173786 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:13.180705 (+ 6919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:13.181473 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:13.181479 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:13.181481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:13.181654 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:13.182153 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:13.182158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:13.182162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:13.182531 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:13.183977 (+ 1446us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:13.184092 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 20:22:13.184330 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":751,"thread_start_us":414,"threads_started":1}
I20260929 20:22:13.186229 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:13.132565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49584 (local address 127.27.185.126:33915)
0929 20:22:13.132778 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:13.132788 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:13.136294 (+ 3506us) server_negotiation.cc:407] Connection header received
0929 20:22:13.139023 (+ 2729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:13.139034 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:13.139156 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:13.139255 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:13.149111 (+ 9856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:13.150323 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:13.152295 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:13.152955 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:13.153072 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:13.168855 (+ 15783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:13.168863 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:13.168870 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:13.168929 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:13.173003 (+ 4074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:13.179550 (+ 6547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:13.179556 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:13.179561 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:13.181281 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:13.182073 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:13.182104 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:13.182107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:13.182806 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:13.183253 (+ 447us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:13.184972 (+ 1719us) server_negotiation.cc:299] Negotiation successful
0929 20:22:13.185524 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 20:22:13.189164 18921 tablet.cc:2369] T 00000000000000000000000000000000 P 9780811cd26f4c31aec7db0006201fbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:22:13.191226 18892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d770e76d9ec84a25b51773530e4135ef: Bootstrap starting.
I20260929 20:22:13.193171 18942 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9780811cd26f4c31aec7db0006201fbe [term 1 FOLLOWER]: Refusing update from remote peer f09baeed299641c5be81dcc0783d9fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:22:13.194568 19003 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f09baeed299641c5be81dcc0783d9fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9780811cd26f4c31aec7db0006201fbe" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 35739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:22:13.213773 18892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d770e76d9ec84a25b51773530e4135ef: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:13.223302 18892 log.cc:824] T 00000000000000000000000000000000 P d770e76d9ec84a25b51773530e4135ef: Log is configured to *not* fsync() on all Append() calls
I20260929 20:22:13.226958 18974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9780811cd26f4c31aec7db0006201fbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader f09baeed299641c5be81dcc0783d9fb2. Latest consensus state: current_term: 1 leader_uuid: "f09baeed299641c5be81dcc0783d9fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09baeed299641c5be81dcc0783d9fb2" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33915 } } peers { permanent_uuid: "d770e76d9ec84a25b51773530e4135ef" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 35255 } } peers { permanent_uuid: "9780811cd26f4c31aec7db0006201fbe" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 35739 } } }
I20260929 20:22:13.227281 18974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9780811cd26f4c31aec7db0006201fbe [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:22:13.230481 19013 mvcc.cc:205] Tried to move back new op lower bound from 7334761812748460032 to 7334761812458520576. Current Snapshot: MvccSnapshot[applied={T|T < 7334761812748460032}]
I20260929 20:22:13.234277 18892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d770e76d9ec84a25b51773530e4135ef: No bootstrap required, opened a new log
I20260929 20:22:13.249845 18989 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09baeed299641c5be81dcc0783d9fb2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f09baeed299641c5be81dcc0783d9fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09baeed299641c5be81dcc0783d9fb2" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33915 } } peers { permanent_uuid: "d770e76d9ec84a25b51773530e4135ef" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 35255 } } peers { permanent_uuid: "9780811cd26f4c31aec7db0006201fbe" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 35739 } } }
I20260929 20:22:13.250262 18989 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09baeed299641c5be81dcc0783d9fb2 [sys.catalog]: This master's current role is: LEADER
I20260929 20:22:13.250036 18892 raft_consensus.cc:348] T 00000000000000000000000000000000 P d770e76d9ec84a25b51773530e4135ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09baeed299641c5be81dcc0783d9fb2" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33915 } } peers { permanent_uuid: "d770e76d9ec84a25b51773530e4135ef" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 35255 } } peers { permanent_uuid: "9780811cd26f4c31aec7db0006201fbe" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 35739 } }
I20260929 20:22:13.250748 18892 raft_consensus.cc:729] T 00000000000000000000000000000000 P d770e76d9ec84a25b51773530e4135ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d770e76d9ec84a25b51773530e4135ef, State: Initialized, Role: FOLLOWER
I20260929 20:22:13.251371 18892 consensus_queue.cc:260] T 00000000000000000000000000000000 P d770e76d9ec84a25b51773530e4135ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09baeed299641c5be81dcc0783d9fb2" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33915 } } peers { permanent_uuid: "d770e76d9ec84a25b51773530e4135ef" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 35255 } } peers { permanent_uuid: "9780811cd26f4c31aec7db0006201fbe" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 35739 } }
I20260929 20:22:13.252080 18974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9780811cd26f4c31aec7db0006201fbe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f09baeed299641c5be81dcc0783d9fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09baeed299641c5be81dcc0783d9fb2" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33915 } } peers { permanent_uuid: "d770e76d9ec84a25b51773530e4135ef" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 35255 } } peers { permanent_uuid: "9780811cd26f4c31aec7db0006201fbe" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 35739 } } }
I20260929 20:22:13.252347 18974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9780811cd26f4c31aec7db0006201fbe [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:22:13.257104 19007 catalog_manager.cc:1347] Generated new cluster ID: 2dbee5873da5453db247206d1247f0b3
I20260929 20:22:13.257243 19007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:22:13.262415 19019 sys_catalog.cc:455] T 00000000000000000000000000000000 P d770e76d9ec84a25b51773530e4135ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09baeed299641c5be81dcc0783d9fb2" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33915 } } peers { permanent_uuid: "d770e76d9ec84a25b51773530e4135ef" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 35255 } } peers { permanent_uuid: "9780811cd26f4c31aec7db0006201fbe" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 35739 } } }
I20260929 20:22:13.264742 19019 sys_catalog.cc:458] T 00000000000000000000000000000000 P d770e76d9ec84a25b51773530e4135ef [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:22:13.264604 18892 sys_catalog.cc:564] T 00000000000000000000000000000000 P d770e76d9ec84a25b51773530e4135ef [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:22:13.265064 18866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d770e76d9ec84a25b51773530e4135ef [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:13.269160 18866 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d770e76d9ec84a25b51773530e4135ef [term 1 FOLLOWER]: Refusing update from remote peer f09baeed299641c5be81dcc0783d9fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:22:13.270006 18867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09baeed299641c5be81dcc0783d9fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d770e76d9ec84a25b51773530e4135ef"
I20260929 20:22:13.273825 18867 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d770e76d9ec84a25b51773530e4135ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f09baeed299641c5be81dcc0783d9fb2 in term 1.
I20260929 20:22:13.279632 19003 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f09baeed299641c5be81dcc0783d9fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d770e76d9ec84a25b51773530e4135ef" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 35255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:22:13.318243 19007 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:22:13.320771 19007 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:22:13.333434 19019 sys_catalog.cc:455] T 00000000000000000000000000000000 P d770e76d9ec84a25b51773530e4135ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader f09baeed299641c5be81dcc0783d9fb2. Latest consensus state: current_term: 1 leader_uuid: "f09baeed299641c5be81dcc0783d9fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09baeed299641c5be81dcc0783d9fb2" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33915 } } peers { permanent_uuid: "d770e76d9ec84a25b51773530e4135ef" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 35255 } } peers { permanent_uuid: "9780811cd26f4c31aec7db0006201fbe" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 35739 } } }
I20260929 20:22:13.333770 19019 sys_catalog.cc:458] T 00000000000000000000000000000000 P d770e76d9ec84a25b51773530e4135ef [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:22:13.334172 19028 sys_catalog.cc:455] T 00000000000000000000000000000000 P d770e76d9ec84a25b51773530e4135ef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f09baeed299641c5be81dcc0783d9fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09baeed299641c5be81dcc0783d9fb2" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33915 } } peers { permanent_uuid: "d770e76d9ec84a25b51773530e4135ef" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 35255 } } peers { permanent_uuid: "9780811cd26f4c31aec7db0006201fbe" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 35739 } } }
I20260929 20:22:13.334628 19028 sys_catalog.cc:458] T 00000000000000000000000000000000 P d770e76d9ec84a25b51773530e4135ef [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:22:13.335815 18989 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09baeed299641c5be81dcc0783d9fb2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f09baeed299641c5be81dcc0783d9fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09baeed299641c5be81dcc0783d9fb2" member_type: VOTER last_known_addr { host: "127.27.185.126" port: 33915 } } peers { permanent_uuid: "d770e76d9ec84a25b51773530e4135ef" member_type: VOTER last_known_addr { host: "127.27.185.125" port: 35255 } } peers { permanent_uuid: "9780811cd26f4c31aec7db0006201fbe" member_type: VOTER last_known_addr { host: "127.27.185.124" port: 35739 } } }
I20260929 20:22:13.336115 18989 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09baeed299641c5be81dcc0783d9fb2 [sys.catalog]: This master's current role is: LEADER
I20260929 20:22:13.341667 19034 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 20:22:13.351713 19007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f09baeed299641c5be81dcc0783d9fb2: Generated new TSK 0
I20260929 20:22:13.352710 19007 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 20:22:13.443034 18896 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:22:13.429765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36030 (local address 127.27.185.125:35255)
0929 20:22:13.429968 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:13.429976 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:13.440741 (+ 10765us) server_negotiation.cc:407] Connection header received
0929 20:22:13.441294 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:13.441300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:13.441405 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:13.441482 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:13.442785 (+ 1303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36030: BlockingRecv error: recv got EOF from 127.0.0.1:36030 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 20:22:13.446259 18971 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:22:13.440484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49592 (local address 127.27.185.126:33915)
0929 20:22:13.440712 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:13.440718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:13.444127 (+ 3409us) server_negotiation.cc:407] Connection header received
0929 20:22:13.444394 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:13.444400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:13.444494 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:13.444575 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:13.445992 (+ 1417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49592: BlockingRecv error: recv got EOF from 127.0.0.1:49592 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260929 20:22:13.462698 19012 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:22:13.460241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50482 (local address 127.27.185.124:35739)
0929 20:22:13.460420 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:13.460428 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:13.460946 (+ 518us) server_negotiation.cc:407] Connection header received
0929 20:22:13.461025 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:13.461031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:13.461135 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:13.461235 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:13.462441 (+ 1206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50482: BlockingRecv error: recv got EOF from 127.0.0.1:50482 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 20:22:13.474254 18896 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:22:13.471884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36040 (local address 127.27.185.125:35255)
0929 20:22:13.472068 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:13.472074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:13.472552 (+ 478us) server_negotiation.cc:407] Connection header received
0929 20:22:13.472774 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:13.472781 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:13.472927 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:13.473049 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:13.473958 (+ 909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36040: BlockingRecv error: recv got EOF from 127.0.0.1:36040 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 20:22:13.482124 18896 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:22:13.479616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36046 (local address 127.27.185.125:35255)
0929 20:22:13.479778 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:13.479785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:13.480653 (+ 868us) server_negotiation.cc:407] Connection header received
0929 20:22:13.480841 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:13.480848 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:13.480942 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:13.481073 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:13.481876 (+ 803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36046: BlockingRecv error: recv got EOF from 127.0.0.1:36046 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 20:22:13.492041 18896 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:22:13.489445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36062 (local address 127.27.185.125:35255)
0929 20:22:13.489632 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:13.489639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:13.490211 (+ 572us) server_negotiation.cc:407] Connection header received
0929 20:22:13.490409 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:13.490416 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:13.490544 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:13.490674 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:13.491758 (+ 1084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36062: BlockingRecv error: recv got EOF from 127.0.0.1:36062 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 20:22:13.503942 18896 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:22:13.501503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36070 (local address 127.27.185.125:35255)
0929 20:22:13.501752 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:13.501759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:13.502327 (+ 568us) server_negotiation.cc:407] Connection header received
0929 20:22:13.502533 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:13.502539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:13.502671 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:13.502819 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:13.503682 (+ 863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36070: BlockingRecv error: recv got EOF from 127.0.0.1:36070 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
W20260929 20:22:13.518079 18896 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:22:13.515530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36086 (local address 127.27.185.125:35255)
0929 20:22:13.515712 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:13.515718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:13.516289 (+ 571us) server_negotiation.cc:407] Connection header received
0929 20:22:13.516624 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:13.516630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:13.516776 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:13.516889 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:13.517811 (+ 922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36086: BlockingRecv error: recv got EOF from 127.0.0.1:36086 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 20:22:13.544036 18896 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:22:13.541463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36094 (local address 127.27.185.125:35255)
0929 20:22:13.541667 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:13.541674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:13.542283 (+ 609us) server_negotiation.cc:407] Connection header received
0929 20:22:13.542523 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:13.542529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:13.542655 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:13.542781 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:13.543775 (+ 994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36094: BlockingRecv error: recv got EOF from 127.0.0.1:36094 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260929 20:22:13.583788 18896 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:22:13.581150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36100 (local address 127.27.185.125:35255)
0929 20:22:13.581383 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:13.581391 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:13.582047 (+ 656us) server_negotiation.cc:407] Connection header received
0929 20:22:13.582270 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:13.582275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:13.582392 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:13.582513 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:13.583503 (+ 990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36100: BlockingRecv error: recv got EOF from 127.0.0.1:36100 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260929 20:22:13.653448 18896 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:22:13.651430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36108 (local address 127.27.185.125:35255)
0929 20:22:13.651659 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:13.651666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:13.652016 (+ 350us) server_negotiation.cc:407] Connection header received
0929 20:22:13.652195 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:13.652201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:13.652306 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:13.652453 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:13.653197 (+ 744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36108: BlockingRecv error: recv got EOF from 127.0.0.1:36108 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260929 20:22:13.790789 18896 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:22:13.788894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36120 (local address 127.27.185.125:35255)
0929 20:22:13.789190 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:13.789197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:13.789467 (+ 270us) server_negotiation.cc:407] Connection header received
0929 20:22:13.789628 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:13.789632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:13.789709 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:13.789797 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:13.790516 (+ 719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36120: BlockingRecv error: recv got EOF from 127.0.0.1:36120 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
W20260929 20:22:14.052541 18896 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:22:14.050609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36132 (local address 127.27.185.125:35255)
0929 20:22:14.050849 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:14.050854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:14.051155 (+ 301us) server_negotiation.cc:407] Connection header received
0929 20:22:14.051319 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:14.051323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:14.051457 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:14.051554 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:14.052361 (+ 807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36132: BlockingRecv error: recv got EOF from 127.0.0.1:36132 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 20:22:14.060716 18986 catalog_manager.cc:1259] Loaded cluster ID: 2dbee5873da5453db247206d1247f0b3
I20260929 20:22:14.060822 18986 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9780811cd26f4c31aec7db0006201fbe: loading cluster ID for follower catalog manager: success
I20260929 20:22:14.063331 18986 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9780811cd26f4c31aec7db0006201fbe: acquiring CA information for follower catalog manager: success
I20260929 20:22:14.065284 18986 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9780811cd26f4c31aec7db0006201fbe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 20:22:14.346276 19034 catalog_manager.cc:1259] Loaded cluster ID: 2dbee5873da5453db247206d1247f0b3
I20260929 20:22:14.346412 19034 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d770e76d9ec84a25b51773530e4135ef: loading cluster ID for follower catalog manager: success
I20260929 20:22:14.349480 19034 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d770e76d9ec84a25b51773530e4135ef: acquiring CA information for follower catalog manager: success
I20260929 20:22:14.351696 19034 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d770e76d9ec84a25b51773530e4135ef: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 20:22:14.581478 19100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:14.574004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36134 (local address 127.27.185.125:35255)
0929 20:22:14.574870 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:14.574876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:14.574903 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:22:14.574968 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:14.574971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:14.575119 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:14.575269 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:22:14.575761 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:14.576643 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:14.578092 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:14.578800 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:14.578956 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:14.579860 (+ 904us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:22:14.580688 (+ 828us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:22:14.580863 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:14.580979 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 20:22:14.581080 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":558,"threads_started":1}
I20260929 20:22:14.596977 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:14.588533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50488 (local address 127.27.185.124:35739)
0929 20:22:14.590738 (+ 2205us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:14.590743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:14.590763 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:22:14.590923 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:14.590927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:14.591099 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:14.591303 (+ 204us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:22:14.591885 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:14.592752 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:14.594373 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:14.594816 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:14.594979 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:14.595658 (+ 679us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:22:14.596249 (+ 591us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:22:14.596438 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:14.596535 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 20:22:14.596634 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2023,"thread_start_us":1908,"threads_started":1}
I20260929 20:22:14.626982 19100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:14.604961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36140 (local address 127.27.185.125:35255)
0929 20:22:14.605211 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:14.605221 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:14.619957 (+ 14736us) server_negotiation.cc:407] Connection header received
0929 20:22:14.620230 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:14.620236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:14.620320 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:14.620442 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:22:14.621161 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:14.622385 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:14.624568 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:14.625300 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:14.625396 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:14.626088 (+ 692us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:22:14.626242 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:22:14.626301 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:14.626520 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 20:22:14.626613 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 20:22:14.628741 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:14.620858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50498 (local address 127.27.185.124:35739)
0929 20:22:14.621023 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:14.621047 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:14.621577 (+ 530us) server_negotiation.cc:407] Connection header received
0929 20:22:14.621841 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:14.621847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:14.621947 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:14.622009 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:22:14.622921 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:14.624054 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:14.626230 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:14.626847 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:14.626951 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:14.627923 (+ 972us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:22:14.628058 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:22:14.628124 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:14.628361 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 20:22:14.628438 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 20:22:14.629240 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:14.604613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49608 (local address 127.27.185.126:33915)
0929 20:22:14.617412 (+ 12799us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:14.617421 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:14.620021 (+ 2600us) server_negotiation.cc:407] Connection header received
0929 20:22:14.621499 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:14.621505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:14.621715 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:14.621879 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:22:14.622832 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:14.623939 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:14.626234 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:14.626894 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:14.627000 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:14.628004 (+ 1004us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:22:14.628507 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:22:14.628593 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:14.628857 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 20:22:14.628951 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12602,"thread_start_us":12814,"threads_started":1}
I20260929 20:22:14.635378 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 18748
I20260929 20:22:14.636435 18812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:22:14.758980 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 18748
I20260929 20:22:14.780566 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 18824
I20260929 20:22:14.781682 18887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:22:14.887936 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 18824
I20260929 20:22:14.908988 28389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu pid 18899
I20260929 20:22:14.910101 18962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:22:15.016911 28389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPFKKvj/build/asan/bin/kudu with pid 18899
2026-09-29T20:22:15Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4926 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9894 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (368066 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST